diff --git a/doc/plans/2026-11-07-phase2-reconciler-data.md b/doc/plans/2026-11-07-phase2-reconciler-data.md index ac73b2cfea4..8124091f5ee 100644 --- a/doc/plans/2026-11-07-phase2-reconciler-data.md +++ b/doc/plans/2026-11-07-phase2-reconciler-data.md @@ -173,6 +173,44 @@ Speed: remote environments have no budget yet (#19863). These are the first remo A Daytona budget needs a dedicated remote baseline. Restart recovery is 5–10× the local values. +## Final Daytona wave and Daytona-only crash points (2026-11-08) + +**Wave `w-20261108T083036-s9401`.** Setup: +- controller at master `8475826d2d`; +- signed `image_only` image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:c24059f1…` (content ID `abc82e78…`, revision `838e9bf3d2`); +- 6 tasks; +- faults: K1, K4 (167 s hold, 175 s outage), K2, K1, K1, K4 (280 s hold, 287 s outage). + +All 6 tasks crossed a fault and all 6 passed. The sandbox sweep was clean, and no run breached a budget. + +The earlier attempt, `w-20261108T082021`, never started. A skopeo blob read timed out during image extraction, and the retry refused the half-written directory. This is a harness problem, fixed in #20021. + +**One real violation: I2 `environment_login_probe_lease` (`pid_dead`), 2 records.** +- These are run-less leases. The codex agent's harness verification acquires one before any run is admitted. +- A restart killed the process holding each lease. The lease stayed `active` until the five-minute idle sweep released it. +- That same idle sweep also releases a live login session's sandbox after five idle minutes. +- The fix is in PR fix/login-probe-lease-settle. The kind gets a settle that moves the lease to cleanup only once its stamped holder is proven gone. The idle sweep now covers only legacy rows saved before holders were stamped. + +**Speed.** +- Restart recovery from `ready` to the next provider event: claude p50 23.9 s, codex 4.6 s. +- Codex time from comment to first token on runs that crossed no fault was about 450 s, but boot was about 32 s. +- The wait came before any run sandbox existed. The tasks launched at 09:55:46, while the first run sandboxes appeared at 10:02:43 (417 s later). +- In between, the codex agent's harness verification ran in a fresh Daytona sandbox four times (09:55:46, 09:56:27, 09:59:43, 10:02:13). Each fault restarted it, and runs stay queued until verification completes. +- This is lifecycle admission, not run boot. In a fault-free product it costs one verification per newly hired agent: one sandbox boot plus a hello probe (for comparison, the seed environment probe took 68 s). + +**Crash points `cp-20261108T104102`** (Daytona, codex A1). The run took 8.7 min; it reused the cached image. + +| Point | Outcome | +|---|---| +| `sandbox.create.after_intent_before_provider` | pass, no violations | +| `sandbox.create.after_provider_before_lease` | pass, no violations | +| `environment.lease_release.after_provider_before_lease` | not reached: a plain native Daytona run now releases through teardown (the two teardown points below), not this path | +| `environment.teardown.after_claim_before_destroy` | pass, no violations | +| `environment.teardown.after_destroy_before_lease` | pass, no violations | +| `environment.stop_retain.after_claim_before_stop` | not reached: needs a stop-and-retain intent (a native workspace export resume), which no harness workload produces yet | + +The R6 sweep found no orphaned sandboxes and no held leases. The first crash-point attempt, `cp-20261108T101304`, exited 2 before any point ran. Cause: the same skopeo timeout, plus a per-run image cache that downloaded the image again. #20021 fixes both. + ## Step 2: kinds moved to the shared path Order of the moves, highest risk first: @@ -197,6 +235,7 @@ Phase 3 is not started. It needs a separate go-ahead. - **Daytona** is covered by one wave only (see "Daytona wave"). The first apparent content-ID mismatch was a misreading: CI run 42422571919 built target commit `a37a2e4`, not master. Local content IDs reproduce CI's for the same commit. - The remote, workspace-operation, sandbox and accounting kinds (see "Per-kind data"). +- **Follow-up: `environment.stop_retain.after_claim_before_stop` has no live coverage.** No chaos workload creates a stop-and-retain intent (a native workspace export resume on a sandbox), so the crash point is never reached. A workload that forces an export resume on Daytona would cover it. `environment.lease_release.after_provider_before_lease` is also unreached by native runs, which release through teardown. - Crash points that sit between two durable writes. Random waves hit 20 ms–1 s windows only by luck. The crash-point suite (Phase 5) is the tool for those. ## Reproduce diff --git a/server/src/__tests__/environment-login-probe-lease-ownership.test.ts b/server/src/__tests__/environment-login-probe-lease-ownership.test.ts new file mode 100644 index 00000000000..b0a30400823 --- /dev/null +++ b/server/src/__tests__/environment-login-probe-lease-ownership.test.ts @@ -0,0 +1,143 @@ +import { spawn } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, createDb, environmentLeases, environments } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { currentControllerProcess, currentControllerProcessStamp, currentProcessSpaceHash, readControllerProcess } from "../services/controller-process-identity.js"; +import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../services/environment-lease-controller.js"; +import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js"; +import { settleEnvironmentLoginProbeLease } from "../services/environment-login-probe-lease-settlement.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js"; +import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; +const KIND = "environment_login_probe_lease"; + +async function exitedPid(): Promise { + const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); + const pid = child.pid!; + await new Promise(resolve => { child.once("exit", () => resolve()); child.kill("SIGKILL"); }); + return pid; +} + +suite("environment_login_probe_lease on the shared ownership path", () => { + let db!: ReturnType; + let temporary: Awaited> | null = null; + let companyId = ""; + let environmentId = ""; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-login-probe-lease-ownership-"); + db = createDb(temporary.connectionString); + companyId = randomUUID(); + environmentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "LPL", + requireBoardApprovalForNewAgents: false }); + await db.insert(environments).values({ id: environmentId, name: "Fake Sandbox", driver: "sandbox", + status: "active", config: { provider: "fake", image: "ubuntu:24.04" } }); + }, 30_000); + + afterEach(async () => { vi.unstubAllEnvs(); await db.delete(environmentLeases); }); + afterAll(async () => { await temporary?.cleanup(); }); + + async function lease(holder: Record | null, providerLeaseId?: string) { + const id = randomUUID(); + await db.insert(environmentLeases).values({ id, companyId, environmentId, heartbeatRunId: null, status: "active", + leasePolicy: "ephemeral", provider: "fake", providerLeaseId: providerLeaseId ?? `sandbox://fake/${id}`, + updatedAt: new Date(Date.now() - 60 * 60_000), + metadata: holder ? { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: holder } : {} }); + return id; + } + async function row(id: string) { + const [found] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)); + return found!; + } + async function deadHolder() { + return { bootId: "earlier-boot", pid: await exitedPid(), hostHash: currentProcessSpaceHash(), processStartedAt: null }; + } + const pass = (record = true) => runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record }); + + it("moves a dead holder's lease to cleanup on the next pass, due at once", async () => { + const leaseId = await lease(await deadHolder()); + const report = await pass(); + expect(report.kinds.find(kind => kind.kind === KIND)).toMatchObject({ settled: 1, reasons: { pid_dead: 1 } }); + const after = await row(leaseId); + expect(after).toMatchObject({ status: "pending_cleanup", failureReason: "orphaned_active_lease_recovered" }); + expect(after.metadata?.recoveredRunlessLease).toEqual({ ownerProof: "pid_dead" }); + expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now()); + expect(report.newViolations.filter(violation => violation.kind === KIND)).toEqual([]); + }); + + it("never takes a live holder's lease, however long it has been idle", async () => { + const self = await currentControllerProcess(); + const alive = await lease({ ...self, bootId: "another-boot" }); + const mine = await lease(currentControllerProcessStamp()); + const foreign = await lease({ bootId: "other", pid: 1, hostHash: "another-pid-space", processStartedAt: null }); + const report = await pass(); + expect(report.kinds.find(kind => kind.kind === KIND)).toMatchObject({ settled: 0 }); + expect(await heartbeatService(db).sweepOrphanedActiveLeases({ backoffMs: 0 })).toEqual({ recovered: 0 }); + for (const id of [alive, mine, foreign]) expect((await row(id)).status).toBe("active"); + }); + + it("never settles on a read-only pass", async () => { + const leaseId = await lease(await deadHolder()); + const report = await pass(false); + expect(report.kinds.find(kind => kind.kind === KIND)).toMatchObject({ settled: 0, verdicts: { gone: 1 } }); + expect((await row(leaseId)).status).toBe("active"); + }); + + it("does not settle a lease re-stamped by a new holder after the proof", async () => { + const leaseId = await lease(await deadHolder()); + const { records } = await OWNABLE_KINDS.find(kind => kind.id === KIND)!.listInProgress!(db, 10, 0); + const record = records.find(candidate => candidate.recordId === leaseId)!; + await db.update(environmentLeases).set({ metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: currentControllerProcessStamp() } }) + .where(eq(environmentLeases.id, leaseId)); + expect(await settleEnvironmentLoginProbeLease(db, record, { now: new Date(), proof: { verdict: "gone", reason: "pid_dead" } })) + .toEqual({ settled: false, reason: "owner_changed_since_proof" }); + expect((await row(leaseId)).status).toBe("active"); + }); + + it("keeps a lease whose provider resource another live lease still holds", async () => { + const shared = "sandbox://fake/shared"; + const orphan = await lease(await deadHolder(), shared); + await lease(currentControllerProcessStamp(), shared); + await pass(); + expect((await row(orphan)).status).toBe("active"); + }); + + it("still recovers a legacy lease saved before holders were stamped, after the idle backoff", async () => { + const legacy = await lease(null); + const report = await pass(); + expect(report.kinds.find(kind => kind.kind === KIND)).toMatchObject({ settled: 0 }); + expect(await heartbeatService(db).sweepOrphanedActiveLeases({ backoffMs: 5 * 60_000 })).toEqual({ recovered: 1 }); + expect((await row(legacy)).status).toBe("pending_cleanup"); + }); + + function railway(deploymentId: string) { + vi.stubEnv("RAILWAY_PROJECT_ID", "project"); vi.stubEnv("RAILWAY_ENVIRONMENT_ID", "production"); + vi.stubEnv("RAILWAY_SERVICE_ID", "server"); vi.stubEnv("RAILWAY_DEPLOYMENT_ID", deploymentId); + vi.stubEnv("RAILWAY_VOLUME_ID", "volume"); vi.stubEnv("RAILWAY_VOLUME_MOUNT_PATH", "/data"); + } + + it("stamps the platform deployment so a redeploy can prove the old holder gone", async () => { + railway("deployment-old"); + const stamp = currentControllerProcessStamp(); + expect(stamp.deployment).toEqual({ worker: { provider: "railway", projectId: "project", environmentId: "production", + serviceId: "server", deploymentId: "deployment-old" }, exclusiveVolume: true, volumeId: "volume" }); + expect(readControllerProcess(JSON.parse(JSON.stringify(stamp)))).toEqual(stamp); + const replaced = await lease({ ...stamp, bootId: "old-boot", hostHash: "old-container", pid: 7 }); + const sameDeployment = await lease({ ...stamp, bootId: "sibling-boot", hostHash: "sibling-container", pid: 8 }); + await db.update(environmentLeases).set({ metadata: { leaseController: { ...stamp, bootId: "sibling-boot", hostHash: "sibling-container", pid: 8, + deployment: { ...stamp.deployment, worker: { ...stamp.deployment!.worker, deploymentId: "deployment-new" } } } } }) + .where(eq(environmentLeases.id, sameDeployment)); + railway("deployment-new"); + const report = await pass(); + expect(report.kinds.find(kind => kind.kind === KIND)).toMatchObject({ settled: 1, + reasons: { deployment_replaced: 1, different_process_space: 1 } }); + expect((await row(replaced)).status).toBe("pending_cleanup"); + expect((await row(sameDeployment)).status).toBe("active"); + }); +}); diff --git a/server/src/__tests__/ownership-reconciliation-report.test.ts b/server/src/__tests__/ownership-reconciliation-report.test.ts index f60e77538c7..2a1e0a9457a 100644 --- a/server/src/__tests__/ownership-reconciliation-report.test.ts +++ b/server/src/__tests__/ownership-reconciliation-report.test.ts @@ -143,7 +143,7 @@ describe("report-only ownership reconciliation", () => { expect(new Set(names).size).toBe(names.length); expect(names.length).toBeGreaterThanOrEqual(40); expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id)) - .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "issue_execution_lock", "issue_checkout_lock"]); + .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "issue_execution_lock", "issue_checkout_lock"]); expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true); }); diff --git a/server/src/services/controller-process-identity.ts b/server/src/services/controller-process-identity.ts index deec5a00db6..030dceaa295 100644 --- a/server/src/services/controller-process-identity.ts +++ b/server/src/services/controller-process-identity.ts @@ -2,13 +2,42 @@ import { createHash, randomUUID } from "node:crypto"; import { readFileSync, readlinkSync } from "node:fs"; import { hostname, uptime } from "node:os"; import { readProcessStartedAt } from "./hot-restart.js"; +import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js"; /** * Identity of the server process that started a provider side effect, saved * with its durable record. After a restart, recovery uses it to prove that the * starter has exited, so the side effect cannot still be in flight there. */ -export type ControllerProcess = { bootId: string; pid: number; hostHash: string; processStartedAt: string | null }; +export type ControllerDeployment = { worker: WorkspaceWorker; exclusiveVolume: boolean; volumeId: string | null }; +export type ControllerProcess = { + bootId: string; pid: number; hostHash: string; processStartedAt: string | null; deployment?: ControllerDeployment; +}; + +export function currentControllerDeployment(env: NodeJS.ProcessEnv = process.env): ControllerDeployment | null { + const worker = currentWorkspaceWorker(env); + if (!worker) return null; + const volumeId = env.RAILWAY_VOLUME_ID?.trim() || null; + return { worker, exclusiveVolume: Boolean(volumeId && env.RAILWAY_VOLUME_MOUNT_PATH?.trim()), volumeId }; +} + +function deploymentField(): { deployment?: ControllerDeployment } { + const deployment = currentControllerDeployment(); + return deployment ? { deployment } : {}; +} + +function readDeployment(value: unknown): ControllerDeployment | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const record = value as Record; + const worker = record.worker as Record | null | undefined; + if (!worker || typeof worker !== "object" || worker.provider !== "railway") return null; + const ids = [worker.projectId, worker.environmentId, worker.serviceId, worker.deploymentId]; + if (ids.some(id => typeof id !== "string" || id.length === 0)) return null; + return { worker: { provider: "railway", projectId: ids[0] as string, environmentId: ids[1] as string, + serviceId: ids[2] as string, deploymentId: ids[3] as string }, + exclusiveVolume: record.exclusiveVolume === true, + volumeId: typeof record.volumeId === "string" ? record.volumeId : null }; +} /** * The one boot id of this server process. Native coordinator claims @@ -94,7 +123,7 @@ export async function currentControllerProcess(): Promise { const startedAt = await currentProcessStartedAtMs(); const iso = startedAt === null ? null : new Date(startedAt).toISOString(); resolvedProcessStartedAt = iso; - return { bootId, pid: process.pid, hostHash, processStartedAt: iso }; + return { bootId, pid: process.pid, hostHash, processStartedAt: iso, ...deploymentField() }; } /** @@ -104,7 +133,7 @@ export async function currentControllerProcess(): Promise { * a live PID then stays ambiguous, which fails closed. */ export function currentControllerProcessStamp(): ControllerProcess { - return { bootId, pid: process.pid, hostHash, processStartedAt: resolvedProcessStartedAt }; + return { bootId, pid: process.pid, hostHash, processStartedAt: resolvedProcessStartedAt, ...deploymentField() }; } export function readControllerProcess(value: unknown): ControllerProcess | null { @@ -112,8 +141,10 @@ export function readControllerProcess(value: unknown): ControllerProcess | null const record = value as Record; if (typeof record.bootId !== "string" || typeof record.hostHash !== "string" || typeof record.pid !== "number" || !Number.isInteger(record.pid) || record.pid <= 0) return null; + const deployment = readDeployment(record.deployment); return { bootId: record.bootId, pid: record.pid, hostHash: record.hostHash, - processStartedAt: typeof record.processStartedAt === "string" ? record.processStartedAt : null }; + processStartedAt: typeof record.processStartedAt === "string" ? record.processStartedAt : null, + ...(deployment ? { deployment } : {}) }; } /** diff --git a/server/src/services/environment-login-probe-lease-settlement.ts b/server/src/services/environment-login-probe-lease-settlement.ts new file mode 100644 index 00000000000..4328975e27f --- /dev/null +++ b/server/src/services/environment-login-probe-lease-settlement.ts @@ -0,0 +1,39 @@ +import { and, eq, inArray, isNull, ne, sql } from "drizzle-orm"; +import { environmentLeases, type Db } from "@paperclipai/db"; +import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; +import { pendingCleanupMetadataObjectSql } from "./environment-cleanup-state.js"; +import { holderUnchanged, ORPHANED_RUN_LEASE_FAILURE_REASON } from "./environment-run-lease-settlement.js"; +import { allowLegacyShutdownWorkspaceCleanup } from "./legacy-shutdown-workspace-settlement.js"; +import type { OwnerProof } from "./owner-proof.js"; +import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js"; + +export async function settleEnvironmentLoginProbeLease(db: Db, record: InProgressRecord, + context: { now: Date; proof: OwnerProof }): Promise { + if (record.owner.kind !== "process") return { settled: false, reason: "owner_not_provable" }; + return db.transaction(async transaction => { + const tx = transaction as unknown as Db; + const [lease] = await tx.select().from(environmentLeases) + .where(and(eq(environmentLeases.id, record.recordId), eq(environmentLeases.status, "active"), + isNull(environmentLeases.heartbeatRunId))) + .for("update"); + if (!lease) return { settled: false, reason: "lease_no_longer_active" }; + if (!holderUnchanged(lease, record)) return { settled: false, reason: "owner_changed_since_proof" }; + if (!(await allowLegacyShutdownWorkspaceCleanup(tx, lease))) return { settled: false, reason: "legacy_shutdown_fence" }; + if (lease.provider && lease.providerLeaseId) { + const [other] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId), + inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]))).limit(1); + if (other) return { settled: false, reason: "provider_resource_shared" }; + } + const flipped = await writeEnvironmentCleanup(tx, write => write.update(environmentLeases) + .set({ status: "pending_cleanup", failureReason: ORPHANED_RUN_LEASE_FAILURE_REASON, + metadata: sql`${pendingCleanupMetadataObjectSql()} || ${JSON.stringify({ + pendingCleanupRecoveredAtMs: context.now.getTime(), recoveredRunlessLease: { ownerProof: context.proof.reason } })}::jsonb` }) + .where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"), + isNull(environmentLeases.heartbeatRunId), + sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`)) + .returning({ id: environmentLeases.id }), { companyId: lease.companyId, leaseId: lease.id }); + return flipped.length ? { settled: true, reason: context.proof.reason } : { settled: false, reason: "lease_no_longer_active" }; + }); +} diff --git a/server/src/services/environment-run-lease-settlement.ts b/server/src/services/environment-run-lease-settlement.ts index f6020a1ed6f..cbbe4e87367 100644 --- a/server/src/services/environment-run-lease-settlement.ts +++ b/server/src/services/environment-run-lease-settlement.ts @@ -91,7 +91,7 @@ export async function settleEnvironmentRunLease(db: Db, record: InProgressRecord /** The proof judged a holder stamp (or its absence, for a legacy row); a lease * re-stamped since then belongs to someone else. */ -function holderUnchanged(lease: LeaseRow, record: InProgressRecord): boolean { +export function holderUnchanged(lease: LeaseRow, record: InProgressRecord): boolean { const holder = readEnvironmentLeaseController(lease.metadata); if (record.owner.kind === "run") return holder === null; if (record.owner.kind !== "process") return false; diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index cc442890675..71a86bd8c7a 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1,5 +1,6 @@ import { settleOwnableKind } from "../recovery/ownership-invariant-report.js"; import { ORPHANED_RUN_LEASE_FAILURE_REASON } from "../environment-run-lease-settlement.js"; +import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../environment-lease-controller.js"; import { readNativeStartupCleanup, readNativeStartupContext } from "../native-runtime/native-startup-cleanup.js"; import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js"; import { pendingFailedNativeRetryCancellationCondition } from "../native-runtime/native-cancellation-request.js"; @@ -1561,8 +1562,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD // whose native workspace export is still pending is preserved, never // released, while its run is not executing here. // - // A login or probe lease has no run. It keeps the idle backoff below until - // its kind moves to the shared path. + // A run-less (login, probe, harness verification) lease is settled by the + // shared ownership path once its stamped holder is proven gone. Only a + // legacy row saved before holders were stamped keeps the idle backoff below. async function sweepOrphanedActiveLeases(opts: { /** Idle time before a run-less (login/probe) lease is recovered. */ backoffMs: number; @@ -1624,6 +1626,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD opts.companyId ? eq(environmentLeases.companyId, opts.companyId) : undefined, opts.leaseId ? eq(environmentLeases.id, opts.leaseId) : undefined, isNull(environmentLeases.heartbeatRunId), + sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? ${ENVIRONMENT_LEASE_CONTROLLER_KEY})`, lte(environmentLeases.updatedAt, cutoff), )) .orderBy(asc(environmentLeases.updatedAt)) diff --git a/server/src/services/owner-proof.ts b/server/src/services/owner-proof.ts index f190bdc9eda..f76dc42bad5 100644 --- a/server/src/services/owner-proof.ts +++ b/server/src/services/owner-proof.ts @@ -1,6 +1,7 @@ import { hostname } from "node:os"; import { currentControllerBootId, + currentControllerDeployment, currentControllerProcessStamp, currentProcessSpaceHash, observedProcessStartedAt, @@ -9,7 +10,7 @@ import { PROCESS_START_PRECISION_MS, UNKNOWN_PROCESS_SPACE, } from "./controller-process-identity.js"; -import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js"; +import type { WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js"; import { ephemeralServerEnabled } from "./ephemeral-server-mode.js"; /** `processSpace` value written when a board operator confirmed that this exact @@ -151,10 +152,7 @@ function processGroupExists(processGroupId: number): boolean { * the previous deployment before the new one starts, so the two never overlap. * A service with a volume cannot have replicas. */ export function currentDeploymentIdentity(env: NodeJS.ProcessEnv = process.env): ProcessProbe["deployment"] { - const worker = currentWorkspaceWorker(env); - if (!worker) return null; - const volumeId = env.RAILWAY_VOLUME_ID?.trim() || null; - return { worker, exclusiveVolume: Boolean(volumeId && env.RAILWAY_VOLUME_MOUNT_PATH?.trim()), volumeId }; + return currentControllerDeployment(env); } export function currentProcessProbe(): ProcessProbe { diff --git a/server/src/services/recovery/ownership-registry.ts b/server/src/services/recovery/ownership-registry.ts index f1ab189929f..0dfdf3dab45 100644 --- a/server/src/services/recovery/ownership-registry.ts +++ b/server/src/services/recovery/ownership-registry.ts @@ -54,6 +54,7 @@ import { pendingCleanupDeadlineSql, recoveredOrphanFirstAttemptSql, } from "../environment-cleanup-state.js"; +import { settleEnvironmentLoginProbeLease } from "../environment-login-probe-lease-settlement.js"; import { settleEnvironmentRunLease } from "../environment-run-lease-settlement.js"; import { settleEnvironmentTeardownClaim } from "../environment-teardown-claim-settlement.js"; import { settleSandboxCreationIntent } from "../sandbox-creation-intent-settlement.js"; @@ -429,6 +430,8 @@ function controllerStamp(value: unknown, scope: string): RecordOwner { return { kind: "process", stamp: { kind: "local", scope, generation: null, host: null, processSpace: controller.hostHash, bootId: controller.bootId, pid: controller.pid, processStartedAt: controller.processStartedAt, processGroupId: null, + ...(controller.deployment ? { worker: controller.deployment.worker, exclusiveVolume: controller.deployment.exclusiveVolume, + volumeId: controller.deployment.volumeId } : {}), } }; } @@ -706,11 +709,12 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [ nextDueAt: nextRetainedOrphanDueAt, }, spoolKind("sandbox_orphan_cleanup_spool", "Orphan cleanup buffer/spool", "ownership", idleOrphanSpoolPath), - leaseKind("environment_login_probe_lease", "Login/probe lease", "ownership", - and(eq(environmentLeases.status, "active"), isNull(environmentLeases.heartbeatRunId)), - // Stamped by the write that makes the lease active. A lease saved before - // the stamp existed names no owner; it ends with its login session. - row => controllerStamp(json(row.metadata)[ENVIRONMENT_LEASE_CONTROLLER_KEY], "environment_login_probe_lease")), + { + ...leaseKind("environment_login_probe_lease", "Login/probe lease", "ownership", + and(eq(environmentLeases.status, "active"), isNull(environmentLeases.heartbeatRunId)), + row => controllerStamp(json(row.metadata)[ENVIRONMENT_LEASE_CONTROLLER_KEY], "environment_login_probe_lease")), + settle: settleEnvironmentLoginProbeLease, + }, // Workspaces { diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 5c14c5b3418..d2a36b3962f 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -280,11 +280,12 @@ workload or `--nth` that does) and `harness_error`. Exit code 0, 1, 3 and 2. (`node scripts/crash-points.mjs --list --env daytona`). The run needs `DAYTONA_API_KEY` and an image matching this checkout (`--daytona-image` or `PAPERCLIP_E2E_DAYTONA_IMAGE`; see "Daytona" above), and refuses to start -without them. A plain task reaches both `sandbox.create.*` points and -`environment.lease_release.after_provider_before_lease`. The other three print -what they need (`needs:` in `--list` and in a `not_reached` result): a failed -release (teardown claim), a reusable lease (reusable destroy), or a -stop-and-retain export. +without them. A plain codex task reaches both `sandbox.create.*` points and +both `environment.teardown.*` points (a native run releases its sandbox through +teardown). `environment.lease_release.after_provider_before_lease` (the plain +release path) and `environment.stop_retain.after_claim_before_stop` (a +stop-and-retain export) were not reached by any workload on 2026-11-08; a +`not_reached` result prints what each needs. **CI.** `.github/workflows/crash-points.yml` runs the same command on demand (`workflow_dispatch`, inputs `points`, `kinds`, `adapters`, `workloads`, `nth`).