diff --git a/doc/plans/2026-11-07-ownership-inventory.md b/doc/plans/2026-11-07-ownership-inventory.md index 4b205b55504..bb2ea28dfc7 100644 --- a/doc/plans/2026-11-07-ownership-inventory.md +++ b/doc/plans/2026-11-07-ownership-inventory.md @@ -346,7 +346,7 @@ Scope: workspace operations (#19807 intents), runtime leases, runtime services, | Managed runtime-control operation (start/stop/restart/repair/run) | `workspace_operations.status` (+ `metadata.runtimeControlOwner`) | workspace-operations.ts:559-578 (stamp :539-550, 10 s heartbeat :586-612) | `running` | `{ownerId: random UUID per process, pid, heartbeatAt}`. No host, boot id, or process start time (:25-29, :69-74) | Owner settle :671-690, owner failure :701-728, own deadline 30 min / 4 h (:46, :53, :656-668). Sweep `sweepRuntimeControlOperations` :297-415, run at startup (index.ts:1087) and before each control (:448-454, routes/execution-workspaces.ts:484, :570) | Same ownerId and not in the in-process set → abandoned (:334-335). Otherwise `process.kill(pid,0)` (:126-135) or **TTL/age**: 60 s without a heartbeat (:38, :346). CAS on `updated_at` (:142-147, :379-383) | ownership | 1 (no host/boot), 2 | A, C | | Native workspace finalize attempt (copyback export) | `workspace_operations.status`, phase `workspace_finalize`, `metadata.owningService='native_workspace_finalizer'` | native-runtime/native-workspace-finalizer.ts:170 → workspace-operations.ts:559 | `running` (`pending` is also read as unresolved) | none on the row. Run id only. Physical owner is the `nativeWorkspaceFinalizationOwner` receipt (row below) | (a) Normal settle workspace-operations.ts:671-728. (b) #19807 supersession native-workspace-finalization-supersession.ts:21-91, called from native-workspace-finalizer.ts:94. (c) Offline maintenance offline-workspace-operation-reconciliation.ts:30-118, called from maintenance/ephemeral-migration-data.ts:52. (d) Online variant :152-157 (**no production caller**) | (b) Caller holds finalization ownership, plus the in-process registry `isNativeWorkspaceFinalizationOperationActive` (workspace-operations.ts:114-119) and a later terminal attempt. (c) Advisory lock :41, committed coordinator with no lease :52-57, finalized sync hash :58-60, accepted result :65-70, later succeeded receipt :83-92. Never age | workspace-preservation | 1, 2 | B, C | | Remote provision command intent | `workspace_operations.status` + `metadata.uncertain`, phase `remote_workspace_provision` | remote-workspaces.ts:973-975 (`running`, `uncertain:true`) | `running` / `uncertain=true` | `heartbeatRunId` + `metadata.epoch` (fenced by authority epoch/writerRunId :964) | Success only: remote-workspaces.ts:989-990. On a non-zero exit, a timeout (:980-982), or a crash, **no write settles the row** (grep finds no other writer) | none. Reservation for any other run is refused with `remote_workspace_provision_unverified` (:220-225). Offline migration counts it as a blocker (remote-workspace-migration-tree.ts:160, remote-workspace-migration.ts:237) | workspace-preservation | 1, 2, 3, 7 | B, C | -| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). No owner-death proof | workspace-preservation | 1, 2 | B, C | +| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). A quiesced save whose run is terminal is settled `failed` by the ownership reconciler (`settleAbandonedRemoteSnapshotSave`) | workspace-preservation | 1, 2 | B, C | | Remote workspace migration op | `workspace_operations.status`, phase `remote_workspace_migration` | remote-workspace-migration.ts:375 | `running` | none (workspace id) | Explicit publication or repair by migration (unverified line). The runtime-control sweep skips it on purpose (workspace-operations.ts:319-323) | none | workspace-preservation | 1, 2 | B, C | | Execution workspace runtime lease | `execution_workspace_runtime_leases` (row existence) | workspace-runtime-leases.ts:194-209 (create), :235-247 (renew), :255-273 (reclaim). Called only from routes/execution-workspaces.ts:567 | row present | `owner_key` `issue:`/`run:`/`agent:` (+ FK columns). No process identity | `release` :291-313. In production it is called only with `force` on archive (routes/execution-workspaces.ts:1270). Otherwise the lease is reclaimed lazily at the next claim (:250-283) | `evaluateStaleOwner` :85-119: issue hidden or terminal, run terminal, or **TTL/age** 30 min (:39, :90). An agent-scoped owner has TTL only (:116-118) | ownership | 1, 2 | A, C | | Workspace runtime service (local process) | `workspace_runtime_services.status` | workspace-runtime.ts (insert line unverified) | `provisioning`, `starting`, `running` | `startedByRunId`, `ownerAgentId`. The pid lives in a local service registry file, not on the row | Startup `reconcilePersistedRuntimeServicesOnStartup` workspace-runtime.ts:8636, which adopts or stops each service (index.ts:1101, not awaited) | Registry record match (workspace-runtime.ts:8781-8797) plus `isPidAlive` local-service-supervisor.ts:218, :245-250. Same host only | ownership | 1, 2 | A, C | diff --git a/server/src/__tests__/ownership-reconciliation-report.test.ts b/server/src/__tests__/ownership-reconciliation-report.test.ts index 29708251964..82489deaa92 100644 --- a/server/src/__tests__/ownership-reconciliation-report.test.ts +++ b/server/src/__tests__/ownership-reconciliation-report.test.ts @@ -145,7 +145,7 @@ describe("report-only ownership reconciliation", () => { expect(new Set(names).size).toBe(names.length); expect(names.length).toBeGreaterThanOrEqual(40); expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id)) - .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]); + .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]); expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true); }); @@ -371,6 +371,43 @@ describe("report-only ownership reconciliation", () => { expect(doneRun.status).toBe("succeeded"); }); + it("settles a remote snapshot save that stopped without finishing after its run ended, and keeps open or live ones", async () => { + const run = (id: string, status: string) => ({ id, companyId, agentId, status, invocationSource: "manual", + ...(status === "running" ? {} : { finishedAt: minutesAgo(2) }) }); + const runs = { failed: randomUUID(), live: randomUUID() }; + await db.insert(heartbeatRuns).values([run(runs.failed, "failed"), run(runs.live, "running")]); + const quiesced = { schema: "paperclip.remote-workspace-save-work.v1", activeIds: [], unknown: false, quiescedAt: minutesAgo(30).toISOString() }; + const ops = { abandoned: randomUUID(), pendingAbandoned: randomUUID(), liveRun: randomUUID(), activeInvocation: randomUUID(), + unknownWork: randomUUID(), notQuiesced: randomUUID(), invalidWork: randomUUID() }; + const op = (id: string, runId: string, saveWork: Record, status = "running") => ({ id, companyId, heartbeatRunId: runId, + phase: "remote_workspace_snapshot", status, + metadata: { epoch: 8, boundary: "final", generation: 9, saveWork, exportCleanup: { state: "pending" } } }); + await db.insert(workspaceOperations).values([ + op(ops.abandoned, runs.failed, quiesced), + op(ops.pendingAbandoned, runs.failed, quiesced, "pending"), + op(ops.liveRun, runs.live, quiesced), + op(ops.activeInvocation, runs.failed, { ...quiesced, activeIds: [randomUUID()], quiescedAt: null }), + op(ops.unknownWork, runs.failed, { ...quiesced, unknown: true }), + op(ops.notQuiesced, runs.failed, { ...quiesced, quiescedAt: null }), + op(ops.invalidWork, runs.failed, { ...quiesced, quiescedAt: "" }), + ]); + const statuses = async () => Object.fromEntries((await db.select({ id: workspaceOperations.id, status: workspaceOperations.status }) + .from(workspaceOperations).where(eq(workspaceOperations.phase, "remote_workspace_snapshot"))) + .map(row => [Object.keys(ops).find(name => ops[name as keyof typeof ops] === row.id), row.status])); + + const first = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() }); + expect(kindReport(first, "remote_snapshot_save")).toMatchObject({ inProgress: 7, settled: 2 }); + expect(await statuses()).toEqual({ abandoned: "failed", pendingAbandoned: "failed", liveRun: "running", activeInvocation: "running", + unknownWork: "running", notQuiesced: "running", invalidWork: "running" }); + const [settled] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, ops.abandoned)); + expect(settled.metadata).toMatchObject({ saveWork: quiesced, exportCleanup: { state: "pending" }, + abandonedSave: { reason: "abandoned_remote_snapshot_save", runId: runs.failed, previousStatus: "running" }, + lastFailure: { code: "remote_workspace_save_abandoned" } }); + expect(settled.finishedAt).not.toBeNull(); + const [failedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runs.failed)); + expect(failedRun.status).toBe("failed"); + }); + it("settles a workspace command whose process died mid-command, and keeps live or unprovable ones", async () => { const run = (id: string, status: string) => ({ id, companyId, agentId, status, invocationSource: "manual", ...(status === "running" ? {} : { finishedAt: minutesAgo(2) }) }); diff --git a/server/src/services/recovery/ownership-registry.ts b/server/src/services/recovery/ownership-registry.ts index 052bdd83ddc..00bb3a345bf 100644 --- a/server/src/services/recovery/ownership-registry.ts +++ b/server/src/services/recovery/ownership-registry.ts @@ -36,7 +36,7 @@ import { settleNativeAcceptedResultLease, } from "../native-runtime/native-accepted-result-lease-settlement.js"; import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js"; -import { settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js"; +import { settleAbandonedRemoteSnapshotSave, settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js"; import { LOCAL_WORKSPACE_COMMAND_PHASES, runtimeControlOperationOwnerStamp, settleRuntimeControlOperation } from "../workspace-operations.js"; import { copybackOwnerHoldOwner, settleCopybackOwnerHold } from "./copyback-owner-hold.js"; import { applyObservedFenceProviderProof, observedWorkspaceFenceOwner, settleObservedWorkspaceFence } from "./observed-workspace-fence.js"; @@ -866,9 +866,9 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [ and(eq(workspaceOperations.phase, "remote_workspace_provision"), or(eq(workspaceOperations.status, "running"), sql`(${workspaceOperations.metadata}->>'uncertain')::boolean is true`)), row => runOwner(row.heartbeatRunId)), - operationKind("remote_snapshot_save", "Remote snapshot save", "workspace_preservation", + { ...operationKind("remote_snapshot_save", "Remote snapshot save", "workspace_preservation", and(eq(workspaceOperations.phase, "remote_workspace_snapshot"), inArray(workspaceOperations.status, ["running", "pending"])), - row => runOwner(row.heartbeatRunId)), + row => runOwner(row.heartbeatRunId)), settle: settleAbandonedRemoteSnapshotSave }, operationKind("remote_workspace_migration", "Remote workspace migration op", "workspace_preservation", and(eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "running")), () => ({ kind: "none" })), diff --git a/server/src/services/recovery/workspace-finalize-attempt-settlement.ts b/server/src/services/recovery/workspace-finalize-attempt-settlement.ts index 73e800bfa1a..f6f70c99440 100644 --- a/server/src/services/recovery/workspace-finalize-attempt-settlement.ts +++ b/server/src/services/recovery/workspace-finalize-attempt-settlement.ts @@ -1,8 +1,11 @@ -import { and, eq, inArray } from "drizzle-orm"; -import { heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db"; import { persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; import type { OwnerProof } from "../owner-proof.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; +import { readSaveWork } from "../remote-workspaces.js"; import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js"; +import { RETAINED_SOURCE_FALLBACK_CHECK_KEY } from "../workspace-recovery-fallback-discovery.js"; import { LOCAL_WORKSPACE_COMMAND_PHASES, isOwnerStampedWorkspaceOperationLive } from "../workspace-operations.js"; import type { InProgressRecord, SettleOutcome } from "./ownership-registry.js"; @@ -103,3 +106,64 @@ function abandonedWorkspaceOperationSettle(spec: { phases: readonly string[]; st return outcome; }; } + +export const ABANDONED_REMOTE_SNAPSHOT_SAVE_REASON = "abandoned_remote_snapshot_save"; + +export async function settleAbandonedRemoteSnapshotSave(db: Db, record: InProgressRecord, + context: { now: Date; proof: OwnerProof }): Promise { + let publication: ActivityPublication | undefined; + const outcome = await db.transaction(async (transaction): Promise => { + const tx = transaction as unknown as Db; + const [operation] = await tx.select().from(workspaceOperations) + .where(and(eq(workspaceOperations.id, record.recordId), eq(workspaceOperations.phase, "remote_workspace_snapshot"), + inArray(workspaceOperations.status, ["running", "pending"]))) + .for("update"); + if (!operation) return { settled: false, reason: "attempt_no_longer_in_progress" }; + const metadata = operation.metadata ?? {}; + const work = readSaveWork(metadata); + if (!work || work.unknown || work.activeIds.length > 0 || !work.quiescedAt) return { settled: false, reason: "save_work_unsettled" }; + const [run] = operation.heartbeatRunId ? await tx.select({ status: heartbeatRuns.status }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, operation.heartbeatRunId), inArray(heartbeatRuns.status, TERMINAL_RUN_STATUSES))) : []; + if (!run) return { settled: false, reason: "owner_run_active" }; + const now = context.now; + const provenance = { + schema: "paperclip.remote-snapshot-save-abandonment.v1", + reason: ABANDONED_REMOTE_SNAPSHOT_SAVE_REASON, + proof: context.proof.reason, + previousStatus: operation.status, + runId: operation.heartbeatRunId, + quiescedAt: work.quiescedAt, + reconciledAt: now.toISOString(), + }; + const workKey = operation.executionWorkspaceId ? `workspace:${operation.executionWorkspaceId}` + : operation.issueId ? `issue:${operation.issueId}` : null; + if (workKey) await notifyTerminalWorkspaceWork(tx, workKey); + const [row] = await tx.update(workspaceOperations).set({ + status: "failed", exitCode: 1, finishedAt: now, updatedAt: now, + stderrExcerpt: `${operation.stderrExcerpt ?? ""}The workspace save stopped without finishing, and its run has ended.\n`.slice(-4096), + metadata: { ...metadata, abandonedSave: provenance, + lastFailure: { code: "remote_workspace_save_abandoned", at: now.toISOString() } }, + }).where(and(eq(workspaceOperations.id, operation.id), eq(workspaceOperations.status, operation.status))) + .returning({ id: workspaceOperations.id }); + if (!row) return { settled: false, reason: "attempt_no_longer_in_progress" }; + if (operation.executionWorkspaceId) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery); + await tx.update(issueRecoveryActions).set({ + evidence: sql`${issueRecoveryActions.evidence} - ${RETAINED_SOURCE_FALLBACK_CHECK_KEY}::text`, + }).where(and(eq(issueRecoveryActions.companyId, operation.companyId), inArray(issueRecoveryActions.status, ["active", "escalated"]), + sql`${issueRecoveryActions.evidence}->${RETAINED_SOURCE_FALLBACK_CHECK_KEY}->>'reason' = 'workspace_operation_running'`, + sql`exists (select 1 from heartbeat_runs source where source.company_id = ${issueRecoveryActions.companyId} + and source.id::text = ${issueRecoveryActions.evidence}->>'runId' + and source.runner_profile_json#>>'{remoteWorkspace,workspaceId}' = ${operation.executionWorkspaceId})`)); + } + const activity = await persistActivity(tx, { + companyId: operation.companyId, actorType: "system", actorId: "ownership-reconciler", + action: "workspace_operation.reconciled", entityType: "workspace_operation", entityId: operation.id, + runId: operation.heartbeatRunId, issueId: operation.issueId, details: provenance, + }); + publication = activity.publication; + return { settled: true, reason: context.proof.reason }; + }); + if (outcome.settled && publication) publishActivity(publication); + return outcome; +} diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index c3020f90bb1..ea990eb8fc0 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -69,7 +69,7 @@ type RemoteWorkspaceSaveTiming = RemoteWorkspaceSavePhaseTiming & { }; const RETIRED_SAVE_SCHEMA = "paperclip.remote-workspace-retired-save.v1"; type SaveWork = { schema: typeof SAVE_WORK_SCHEMA; activeIds: string[]; unknown: boolean; quiescedAt: string | null }; -function readSaveWork(metadata: Record | null | undefined): SaveWork | null { +export function readSaveWork(metadata: Record | null | undefined): SaveWork | null { const work = object(metadata?.saveWork); return work.schema === SAVE_WORK_SCHEMA && Array.isArray(work.activeIds) && work.activeIds.every(id => typeof id === "string") && new Set(work.activeIds).size === work.activeIds.length diff --git a/server/src/services/workspace-recovery-fallback.test.ts b/server/src/services/workspace-recovery-fallback.test.ts index 2d79f39aa35..931eb3e42b2 100644 --- a/server/src/services/workspace-recovery-fallback.test.ts +++ b/server/src/services/workspace-recovery-fallback.test.ts @@ -17,6 +17,7 @@ import { prepareRemoteWorkspace, reserveRemoteWorkspace, saveRemoteWorkspace } f import { readSystemRecovery } from "./system-recovery-state.js"; import { reconcileRetainedWorkspaces, recoverRetainedWorkspace, retainedWorkspaceCandidates } from "./workspace-recovery.js"; import { nextWorkspaceRecoveryAt } from "./system-recovery.js"; +import { createOwnershipReportState, runOwnershipReconciliationPass } from "./recovery/ownership-invariant-report.js"; import { RETAINED_SOURCE_FALLBACK_OUTCOME, RETAINED_SOURCE_QUARANTINE_PHASE } from "./workspace-recovery-fallback.js"; import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js"; import { acceptedRemoteExport, action, connectionString, db, run, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js"; @@ -339,6 +340,30 @@ describe("retained source fallback", () => { expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME }); }, 60_000); + it("continues once the reconciler settles a final save a restart left running after its run ended", async () => { + const f = await unverifiableRetainedSource(); + const [save] = await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, + heartbeatRunId: f.runId, issueId: f.issueId, phase: "remote_workspace_snapshot", status: "running", + metadata: { epoch: 8, boundary: "final", generation: 99, exportCleanup: { state: "pending" }, + saveWork: { schema: "paperclip.remote-workspace-save-work.v1", activeIds: [], unknown: false, quiescedAt: new Date(Date.now() - 60_000).toISOString() } }, + }).returning(); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []); + const deferred = await action(f.actionId); + expect(deferred.status).toBe("active"); + expect(deferred.evidence.retainedSourceFallbackCheck).toMatchObject({ reason: "workspace_operation_running" }); + // Twelve refusals spend the check budget: no further check is scheduled. + await db.update(issueRecoveryActions).set({ evidence: { ...deferred.evidence, retainedSourceFallbackCheck: { + ...(deferred.evidence.retainedSourceFallbackCheck as object), checks: 12, nextCheckAt: null } } }).where(eq(issueRecoveryActions.id, f.actionId)); + expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).not.toContain(f.actionId); + + await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() }); + expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, save.id)))[0]) + .toMatchObject({ status: "failed", metadata: { abandonedSave: { reason: "abandoned_remote_snapshot_save" } } }); + expect((await action(f.actionId)).evidence.retainedSourceFallbackCheck).toBeUndefined(); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []); + expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME }); + }, 60_000); + it("settles a stale exhausted hold as recovered when a sibling action saved the run's files", async () => { const f = await acceptedRemoteExport(); await recoverRetainedWorkspace(f.input);