mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
feat(recovery): settle orphaned active environment leases on the shared ownership path (#19964)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs hold environment leases (local workspaces and remote sandboxes) while they execute > - A crash between the terminal run write and the lease release leaves the lease `active` with no holder > - A sweep released such a lease when the run was terminal and the lease was idle for five minutes. The age was the only guard against a race with the live executor, which releases the lease itself with the run's resource disposition > - Phase 2 of the durable ownership plan moves each in-progress record kind onto one shared owner proof (#19933 added the settle hook) > - This pull request moves `environment_run_lease_active` onto that path: the lease names its holder, the shared prover judges the holder, and the settle never destroys a sandbox > - The benefit is that a dead holder's lease settles on the next pass, a live holder is never raced, and the five-minute age gate is gone ## Linked Issues or Issue Description Refs #19933 **What happened?** An orphaned active environment lease was released by age: "run terminal plus five minutes" (the reaper in `heartbeat/recovery.ts` and the live-recovery `orphanLease` worker). The ownership data showed one such lease settle 119 s after it was ready. Without the age gate the sweep could flip a lease to `pending_cleanup` (destroy) while the executor or a recovery path was about to retain it. **Expected behavior** A run lease settles when its holder is proven gone, by identity and not by age. A live holder is never raced. A settle never destroys a sandbox whose files may still be needed. **Steps to reproduce** 1. Start a run that holds an environment lease. 2. Kill the server after the run status becomes terminal and before the lease release. 3. Restart. The lease stays `active` until the five-minute age passes, then the sweep destroys the sandbox. ## What Changed - `environments.ts`: the write that makes a lease active (insert and same-run reacquire) stamps the holder process in `metadata.leaseController`. The stamp does no I/O (`environment-lease-controller.ts`). - `recovery/ownership-registry.ts`: `environment_run_lease_active` lists leases with their run status. The owner is the run while it is active, then the stamped holder. It registers `settle` and `listForRuns`. - `environment-run-lease-settlement.ts` (new): compare-and-swap settle on the exact row (same run, status `active`, same metadata, terminal run). A sandbox lease gets the stop-and-retain intent (stop and keep, never destroy). Other leases move to `pending_cleanup` as before. It keeps a lease that a pending native export, a legacy shutdown fence, a shared provider resource, or an accepted native result still needs. - `recovery/ownership-invariant-report.ts`: `settleOwnableKind` accepts `currentProcessIdle`. A caller that passes every run this process executes in `skipRunIds` may treat a current-process owner of another run as gone (`current_process_idle`). The periodic pass never does this. - `heartbeat/recovery.ts`: `sweepOrphanedActiveLeases` no longer releases run leases by age. It runs native workspace preservation first (a separate obligation), then calls `settleOwnableKind` with this process's executions. Login and probe leases (no run) keep the idle sweep until their own change. - `heartbeat/live-recovery-work.ts`: a run lease is due now; only run-less leases keep the five-minute idle deadline. - `sandbox-stop-and-retain.ts`: an ownership settle records a recovery marker on the intent and keeps `updatedAt`, so the pending-cleanup sweep stops the sandbox in the same tick. - Deleted: the age gate for run leases and the page-order bump for run leases. ## Verification - `cd server && npx vitest run src/__tests__/environment-run-lease-ownership.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on unchanged master sources, green on this branch (tests were written after the code): - stamps the lease holder in the write that makes a lease active - settles a lease whose holder died after the run went terminal, on the next pass (crash-point state: terminal run, dead holder in this PID space) - stops and keeps a dead holder's sandbox instead of destroying it - leaves a current-process lease to this process: settled only when it is not executing the run - keeps the run fact alone for one release on a single-host server - names no owner on an ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`) - the three updated sweep tests (sandbox stopped and kept, not destroyed) - Live-owner counter-cases (green): run still active; another live PID; another PID space; current process while executing; read-only pass (`record: false`) never settles; a lease re-stamped after the proof is not settled; a legacy lease whose run this boot owns is kept. - Also run green: `recovery.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `workspace-recovery-warm-handoff.test.ts`, `environment-service.test.ts`, `heartbeat-pending-cleanup-sweep.test.ts`, `native-environment-settlement-work.test.ts`, `environment-cleanup-work.test.ts`. - Full typecheck is left to CI (local memory limit). ## Risks - Behavior change: an orphaned sandbox lease is now stopped and kept, not destroyed. Workspace preservation and provider auto-delete decide when it goes away. This can keep stopped sandboxes longer than before. - A recovery path on another process can mark a run terminal while the dead executor's stamp is still on the lease. The settle can then win before that path releases. Because the settle only stops and keeps the sandbox, files are not lost; the later release continues from the stop-only intent. - Previous-release rows: a single-host server keeps the run fact for one release (a run this boot owns is excluded). An ephemeral server never releases an unstamped lease; it stays visible as an I1 gap with no operator action yet. - No migration. The stamp lives in existing lease metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
81f12c21ec
commit
cc072e7c4e
14 files changed
+699
-102
No files matched your search
@@ -0,0 +1,288 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
companies,
|
||||
completionContracts,
|
||||
createDb,
|
||||
environmentLeases,
|
||||
environments,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
nativeRunResults,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
currentControllerBootId,
|
||||
currentControllerProcess,
|
||||
currentControllerProcessStamp,
|
||||
currentProcessSpaceHash,
|
||||
} from "../services/controller-process-identity.js";
|
||||
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../services/environment-lease-controller.js";
|
||||
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
|
||||
import { settleEnvironmentRunLease } from "../services/environment-run-lease-settlement.js";
|
||||
import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/heartbeat.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
|
||||
import {
|
||||
createOwnershipReportState,
|
||||
runOwnershipReconciliationPass,
|
||||
settleOwnableKind,
|
||||
} from "../services/recovery/ownership-invariant-report.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const suite = support.supported ? describe : describe.skip;
|
||||
|
||||
/** A PID in this PID space whose process has exited. */
|
||||
async function exitedPid(): Promise<number> {
|
||||
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
|
||||
const pid = child.pid!;
|
||||
await new Promise<void>(resolve => { child.once("exit", () => resolve()); child.kill("SIGKILL"); });
|
||||
return pid;
|
||||
}
|
||||
|
||||
suite("environment_run_lease_active on the shared ownership path", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let companyId = "";
|
||||
let agentId = "";
|
||||
let sandboxEnvironmentId = "";
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("paperclip-environment-run-lease-ownership-");
|
||||
db = createDb(temporary.connectionString);
|
||||
companyId = randomUUID();
|
||||
agentId = randomUUID();
|
||||
sandboxEnvironmentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "ERL",
|
||||
requireBoardApprovalForNewAgents: false });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
|
||||
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
|
||||
await db.insert(environments).values({ id: sandboxEnvironmentId, name: "Fake Sandbox", driver: "sandbox",
|
||||
status: "active", config: { provider: "fake", image: "ubuntu:24.04" } });
|
||||
}, 30_000);
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(heartbeatRuns);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
async function run(status: string, extra: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
||||
const id = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id, companyId, agentId, invocationSource: "on_demand", status,
|
||||
nextEventSeq: 1, ...extra });
|
||||
return id;
|
||||
}
|
||||
|
||||
/** The state a crash between the terminal run write and the lease release
|
||||
* leaves behind: a terminal run whose lease is still active. */
|
||||
async function lease(runId: string, holder: Record<string, unknown> | null, sandbox = false) {
|
||||
const id = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id, companyId, heartbeatRunId: runId, status: "active",
|
||||
leasePolicy: "ephemeral",
|
||||
...(sandbox ? { environmentId: sandboxEnvironmentId, provider: "fake", providerLeaseId: `sandbox://fake/${id}` } : {}),
|
||||
metadata: holder ? { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: holder } : {} });
|
||||
return id;
|
||||
}
|
||||
|
||||
async function row(id: string) {
|
||||
const [found] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, id));
|
||||
return found!;
|
||||
}
|
||||
|
||||
async function deadHolder() {
|
||||
return { bootId: "earlier-boot", pid: await exitedPid(), hostHash: currentProcessSpaceHash(), processStartedAt: null };
|
||||
}
|
||||
|
||||
it("stamps the lease holder in the write that makes a lease active, with no extra I/O", async () => {
|
||||
const runId = await run("running");
|
||||
const acquired = await environmentService(db).acquireLease({ companyId, environmentId: sandboxEnvironmentId,
|
||||
heartbeatRunId: runId, provider: "fake", providerLeaseId: "sandbox://fake/stamped", metadata: { driver: "sandbox" } });
|
||||
expect(acquired.metadata?.[ENVIRONMENT_LEASE_CONTROLLER_KEY]).toEqual(currentControllerProcessStamp());
|
||||
expect(acquired.metadata?.driver).toBe("sandbox");
|
||||
});
|
||||
|
||||
it("settles a lease whose holder died after the run went terminal, on the next pass", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, await deadHolder());
|
||||
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect(report.kinds.find(kind => kind.kind === "environment_run_lease_active")).toMatchObject({
|
||||
settled: 1, reasons: { pid_dead: 1 } });
|
||||
expect(await row(leaseId)).toMatchObject({ status: "pending_cleanup", failureReason: "orphaned_active_lease_recovered" });
|
||||
});
|
||||
|
||||
it("stops and keeps a dead holder's sandbox instead of destroying it", async () => {
|
||||
const runId = await run("succeeded");
|
||||
const leaseId = await lease(runId, await deadHolder(), true);
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
const after = await row(leaseId);
|
||||
expect(after).toMatchObject({ status: "pending_cleanup", failureReason: "sandbox_stop_pending" });
|
||||
expect(after.metadata?.sandboxStopAndRetain).toMatchObject({ leaseId, runId, builtinProvider: "fake",
|
||||
recoveredOrphanedRunLease: true, ownerProof: "pid_dead" });
|
||||
});
|
||||
|
||||
it("stops a recovered orphan at once, even when the lease was used moments ago; a failed stop backs off", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, await deadHolder(), true);
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
|
||||
const stop = vi.fn(async () => { throw new Error("provider unavailable"); });
|
||||
const heartbeat = heartbeatService(db, { environmentRuntime: {
|
||||
retryPendingSandboxTeardown: stop } as unknown as HeartbeatEnvironmentRuntime });
|
||||
await heartbeat.sweepPendingCleanupLeases({ backoffMs: 5 * 60_000 });
|
||||
expect(stop).toHaveBeenCalledOnce();
|
||||
// The failed first attempt counted; the next one waits out the backoff.
|
||||
await heartbeat.sweepPendingCleanupLeases({ backoffMs: 5 * 60_000 });
|
||||
expect(stop).toHaveBeenCalledOnce();
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it("reads one bounded page per recovery call, so kept leases never block the leases behind them", async () => {
|
||||
const foreign = { bootId: "other", pid: 1, hostHash: "another-pid-space", processStartedAt: null };
|
||||
const keptRuns = await Promise.all(Array.from({ length: 100 }, () => run("failed")));
|
||||
await db.insert(environmentLeases).values(keptRuns.map((runId, index) => ({
|
||||
id: `00000000-0000-4000-8000-${String(index).padStart(12, "0")}`, companyId, heartbeatRunId: runId,
|
||||
status: "active", metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: foreign } })));
|
||||
const orphanId = "ffffffff-0000-4000-8000-000000000000";
|
||||
await db.insert(environmentLeases).values({ id: orphanId, companyId, heartbeatRunId: await run("failed"), status: "active",
|
||||
metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: await deadHolder() } });
|
||||
const heartbeat = heartbeatService(db);
|
||||
expect(await heartbeat.sweepOrphanedActiveLeases({ backoffMs: 0 })).toEqual({ recovered: 0 });
|
||||
expect((await row(orphanId)).status).toBe("active");
|
||||
expect(await heartbeat.sweepOrphanedActiveLeases({ backoffMs: 0 })).toEqual({ recovered: 1 });
|
||||
expect((await row(orphanId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it("waits for native settlement only while the lease's environment exists to release it", async () => {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Native settlement" });
|
||||
const contractId = randomUUID();
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "result-worker-v1", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "result-worker-v1", objective: "Finish", criteria: [] },
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
const native = { runtimeMode: "native", nativeIssueId: issueId, issueId, completionContractId: contractId };
|
||||
const kept = await run("succeeded", native);
|
||||
const orphaned = await run("succeeded", native);
|
||||
const goneEnvironmentId = randomUUID();
|
||||
await db.insert(environments).values({ id: goneEnvironmentId, name: "Deleted", driver: "sandbox", status: "active",
|
||||
config: { provider: "fake", image: "ubuntu:24.04" } });
|
||||
for (const runId of [kept, orphaned]) {
|
||||
const resultId = randomUUID();
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId,
|
||||
serverFingerprint: "fixture", schemaStatus: "accepted", resultJson: {}, canonicalSha256: "a".repeat(64) });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "committed", resultId });
|
||||
}
|
||||
const keptLease = await lease(kept, await deadHolder(), true);
|
||||
const orphanedLease = await lease(orphaned, await deadHolder(), true);
|
||||
await db.update(environmentLeases).set({ environmentId: goneEnvironmentId }).where(eq(environmentLeases.id, orphanedLease));
|
||||
await db.delete(environments).where(eq(environments.id, goneEnvironmentId));
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect((await row(keptLease)).status).toBe("active");
|
||||
expect((await row(orphanedLease)).status).toBe("pending_cleanup");
|
||||
await db.delete(nativeRunFinalizations);
|
||||
await db.delete(nativeRunResults);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(completionContracts);
|
||||
await db.delete(issues).where(eq(issues.id, issueId));
|
||||
});
|
||||
|
||||
it("keeps a lease while its run is active, whatever its holder", async () => {
|
||||
const runId = await run("running");
|
||||
const leaseId = await lease(runId, await deadHolder());
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
await settleOwnableKind(db, "environment_run_lease_active", { currentProcessIdle: true, skipRunIds: new Set() });
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
});
|
||||
|
||||
it("keeps a lease whose holder is alive: another live PID, or another PID space", async () => {
|
||||
const runId = await run("failed");
|
||||
const self = await currentControllerProcess();
|
||||
const alive = await lease(runId, { ...self, bootId: "another-boot" });
|
||||
const foreign = await lease(runId, { bootId: "other", pid: 1, hostHash: "another-pid-space", processStartedAt: null });
|
||||
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect(report.kinds.find(kind => kind.kind === "environment_run_lease_active")).toMatchObject({
|
||||
settled: 0, reasons: { pid_alive: 1, different_process_space: 1 } });
|
||||
expect((await row(alive)).status).toBe("active");
|
||||
expect((await row(foreign)).status).toBe("active");
|
||||
});
|
||||
|
||||
it("leaves a current-process lease to this process: settled only when it is not executing the run", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, currentControllerProcessStamp());
|
||||
// The periodic pass cannot see this process's executions.
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
// The executor is still in its finally: it releases the lease itself.
|
||||
expect(await settleOwnableKind(db, "environment_run_lease_active", { runIds: [runId],
|
||||
skipRunIds: new Set([runId]), currentProcessIdle: true })).toEqual([]);
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
// Without the idle claim, the current process stays alive.
|
||||
expect(await settleOwnableKind(db, "environment_run_lease_active", { runIds: [runId] })).toEqual([]);
|
||||
const settled = await settleOwnableKind(db, "environment_run_lease_active", { runIds: [runId],
|
||||
skipRunIds: new Set(), currentProcessIdle: true });
|
||||
expect(settled).toEqual([{ kind: "environment_run_lease_active", recordId: leaseId, runId, reason: "current_process_idle" }]);
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it("never settles on a read-only pass", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, await deadHolder());
|
||||
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
||||
expect(report.kinds.find(kind => kind.kind === "environment_run_lease_active")).toMatchObject({
|
||||
settled: 0, verdicts: { gone: 1 } });
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
});
|
||||
|
||||
it("does not settle a lease re-stamped by a new holder after the proof", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, await deadHolder());
|
||||
const original = (await row(leaseId)).metadata!;
|
||||
const [record] = await OWNABLE_KINDS.find(kind => kind.id === "environment_run_lease_active")!.listForRuns!(db, [runId]);
|
||||
await db.update(environmentLeases).set({ metadata: { ...original,
|
||||
[ENVIRONMENT_LEASE_CONTROLLER_KEY]: currentControllerProcessStamp() } }).where(eq(environmentLeases.id, leaseId));
|
||||
expect(await settleEnvironmentRunLease(db, record!, { now: new Date(), proof: { verdict: "gone", reason: "pid_dead" } }))
|
||||
.toEqual({ settled: false, reason: "owner_changed_since_proof" });
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
});
|
||||
|
||||
describe("leases saved before holders were stamped", () => {
|
||||
it("keeps the run fact alone for one release on a single-host server", async () => {
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, null);
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it("never releases one whose run this boot owns (a hot-restart adoption may be releasing it)", async () => {
|
||||
const runId = await run("failed", { controllerBootId: currentControllerBootId() });
|
||||
const leaseId = await lease(runId, null);
|
||||
await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
// This process knows it is not executing the run: the local path settles it.
|
||||
expect(await settleOwnableKind(db, "environment_run_lease_active", { runIds: [runId],
|
||||
skipRunIds: new Set([randomUUID()]), currentProcessIdle: true })).toHaveLength(1);
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it("names no owner on an ephemeral server, where the host cannot be known", async () => {
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||
const runId = await run("failed");
|
||||
const leaseId = await lease(runId, null);
|
||||
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
|
||||
expect(report.kinds.find(kind => kind.kind === "environment_run_lease_active")).toMatchObject({
|
||||
settled: 0, verdicts: { unprovable: 1 }, reasons: { no_owner_stamp: 1 } });
|
||||
expect(report.violations.some(violation => violation.recordId === leaseId && violation.rule === "I1_owned")).toBe(true);
|
||||
expect((await row(leaseId)).status).toBe("active");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -191,7 +191,9 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
return (await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success" })
|
||||
.where(eq(environmentLeases.id, leaseId)).returning())[0];
|
||||
});
|
||||
const runtime = { destroyRunLease } as unknown as HeartbeatEnvironmentRuntime;
|
||||
const retryPendingSandboxTeardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
|
||||
({ providerLeaseId: lease.providerLeaseId, state: "stopped" }));
|
||||
const runtime = { destroyRunLease, retryPendingSandboxTeardown } as unknown as HeartbeatEnvironmentRuntime;
|
||||
try {
|
||||
const restarted = heartbeatService(db, { environmentRuntime: runtime });
|
||||
await restarted.sweepOrphanedActiveLeases({ backoffMs: 0 });
|
||||
@@ -206,8 +208,17 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
const afterExport = heartbeatService(db, { environmentRuntime: runtime });
|
||||
await afterExport.sweepOrphanedActiveLeases({ backoffMs: 0 });
|
||||
await afterExport.sweepPendingCleanupLeases({ backoffMs: 0 });
|
||||
expect(destroyRunLease).toHaveBeenCalledOnce();
|
||||
await expect(readFile(output)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
if (status === "active") {
|
||||
// The ownership settle stops the orphaned sandbox and keeps it; it
|
||||
// never destroys it (workspace preservation decides that).
|
||||
expect(destroyRunLease).not.toHaveBeenCalled();
|
||||
expect(retryPendingSandboxTeardown).toHaveBeenCalledOnce();
|
||||
expect(await readFile(output, "utf8")).toBe(nonce);
|
||||
expect(await leaseRow(leaseId)).toMatchObject({ status: "released", cleanupStatus: "success" });
|
||||
} else {
|
||||
expect(destroyRunLease).toHaveBeenCalledOnce();
|
||||
await expect(readFile(output)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}
|
||||
} finally {
|
||||
await rm(sandbox, { recursive: true, force: true });
|
||||
}
|
||||
@@ -400,13 +411,13 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
});
|
||||
|
||||
it("test_defers_a_guarded_lease_instead_of_leaving_it_at_the_front_of_the_page", async () => {
|
||||
const { companyId, agentId, environmentId } = await seedCompanyAgentAndEnvironment();
|
||||
const runId = await insertHeartbeatRun({ companyId, agentId, status: "failed" });
|
||||
const { companyId, environmentId } = await seedCompanyAgentAndEnvironment();
|
||||
const sharedProviderLeaseId = "sandbox://fake/shared-resource";
|
||||
// A run-less (login/probe) lease still pages by idle age.
|
||||
const orphanedLeaseId = await insertActiveLease({
|
||||
companyId,
|
||||
environmentId,
|
||||
heartbeatRunId: runId,
|
||||
heartbeatRunId: null,
|
||||
updatedAt: oldEnough(),
|
||||
providerLeaseId: sharedProviderLeaseId,
|
||||
});
|
||||
@@ -520,9 +531,12 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return row ? { ...row, status: "expired" as const } : null;
|
||||
});
|
||||
const retryPendingSandboxTeardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
|
||||
({ providerLeaseId: lease.providerLeaseId, state: "stopped" }));
|
||||
const heartbeat = heartbeatService(db, {
|
||||
environmentRuntime: {
|
||||
destroyRunLease,
|
||||
retryPendingSandboxTeardown,
|
||||
} as unknown as HeartbeatEnvironmentRuntime,
|
||||
});
|
||||
|
||||
@@ -531,9 +545,12 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
// pending_cleanup teardown both run in this one call.
|
||||
await heartbeat.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000 });
|
||||
|
||||
expect(destroyRunLease).toHaveBeenCalledTimes(1);
|
||||
// The orphaned run's sandbox is stopped and kept, never destroyed.
|
||||
expect(destroyRunLease).not.toHaveBeenCalled();
|
||||
expect(retryPendingSandboxTeardown).toHaveBeenCalledTimes(1);
|
||||
const row = await leaseRow(leaseId);
|
||||
expect(row?.status).toBe("expired");
|
||||
expect(row?.status).toBe("released");
|
||||
expect(row?.metadata?.sandboxStopAndRetainReceipt).toMatchObject({ leaseId, method: "builtin.stopLease" });
|
||||
});
|
||||
|
||||
it("test_stops_the_recovered_sandbox_on_the_startup_call", async () => {
|
||||
@@ -556,9 +573,12 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return row ? { ...row, status: "expired" as const } : null;
|
||||
});
|
||||
const retryPendingSandboxTeardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
|
||||
({ providerLeaseId: lease.providerLeaseId, state: "stopped" }));
|
||||
const heartbeat = heartbeatService(db, {
|
||||
environmentRuntime: {
|
||||
destroyRunLease,
|
||||
retryPendingSandboxTeardown,
|
||||
} as unknown as HeartbeatEnvironmentRuntime,
|
||||
});
|
||||
|
||||
@@ -567,9 +587,12 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
|
||||
// and the recovered lease reaches the provider teardown in this one call.
|
||||
await heartbeat.reapOrphanedRuns({ staleThresholdMs: 0 });
|
||||
|
||||
expect(destroyRunLease).toHaveBeenCalledTimes(1);
|
||||
// The orphaned run's sandbox is stopped and kept, never destroyed.
|
||||
expect(destroyRunLease).not.toHaveBeenCalled();
|
||||
expect(retryPendingSandboxTeardown).toHaveBeenCalledTimes(1);
|
||||
const row = await leaseRow(leaseId);
|
||||
expect(row?.status).toBe("expired");
|
||||
expect(row?.status).toBe("released");
|
||||
expect(row?.metadata?.sandboxStopAndRetainReceipt).toMatchObject({ leaseId, method: "builtin.stopLease" });
|
||||
});
|
||||
|
||||
it("test_logs_a_distinct_error_kind_and_no_exception_field_when_the_sweep_fails", async () => {
|
||||
|
||||
@@ -140,8 +140,8 @@ describe("report-only ownership reconciliation", () => {
|
||||
const names = OWNABLE_KINDS.map(kind => kind.id);
|
||||
expect(new Set(names).size).toBe(names.length);
|
||||
expect(names.length).toBeGreaterThanOrEqual(40);
|
||||
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
|
||||
.toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "wake_request_claimed"]);
|
||||
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id)).toEqual(
|
||||
["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "environment_run_lease_active", "wake_request_claimed"]);
|
||||
expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
|
||||
});
|
||||
|
||||
@@ -150,7 +150,7 @@ describe("report-only ownership reconciliation", () => {
|
||||
const before = await snapshot();
|
||||
const state = createOwnershipReportState();
|
||||
|
||||
const first = await runOwnershipReconciliationPass(db, { state });
|
||||
const first = await runOwnershipReconciliationPass(db, { state, settle: false });
|
||||
|
||||
expect(kindReport(first, "native_executor_claim")).toMatchObject({ inProgress: 1,
|
||||
verdicts: { alive: 1, gone: 0, unprovable: 0 }, reasons: { current_process: 1 } });
|
||||
@@ -178,13 +178,13 @@ describe("report-only ownership reconciliation", () => {
|
||||
"I7_scope", "record_cap", "wake_coverage"]);
|
||||
|
||||
expect(first.awaitingSecondProof).toBe(1);
|
||||
const second = await runOwnershipReconciliationPass(db, { state });
|
||||
const second = await runOwnershipReconciliationPass(db, { state, settle: false });
|
||||
expect(rules(second, "environment_run_lease_active")).toEqual(["I2_settled"]);
|
||||
expect(second.newViolations.map(violation => `${violation.rule}:${violation.kind}`)).toEqual(["I2_settled:environment_run_lease_active"]);
|
||||
|
||||
// A stuck record already reported for I2 needs no further pass.
|
||||
expect(second.awaitingSecondProof).toBe(0);
|
||||
expect((await runOwnershipReconciliationPass(db, { state })).awaitingSecondProof).toBe(0);
|
||||
expect((await runOwnershipReconciliationPass(db, { state, settle: false })).awaitingSecondProof).toBe(0);
|
||||
|
||||
// Report-only: nothing was settled, released or retried.
|
||||
expect(await snapshot()).toEqual(before);
|
||||
|
||||
Reference in new issue
Block a user