Files
PaperClipAI/server/src/services/native-runtime/native-workspace-finalization-ownership.ts
T
DottaandClaude Opus 5.5 cbf811cd3b fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.

**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.

**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.

## What Changed

- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.

Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
  - After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:12:23 -06:00

378 lines
23 KiB
TypeScript

import { notifyNativeResultCommit } from "../native-result-commit-notifications.js";
import { notifyNativeWorkspaceFinalization, releaseNativeWorkspaceFinalization } from "../native-workspace-finalization-notifications.js";
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
import { notifyQueuedRunDispatch } from "../run-dispatch-notifications.js";
import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
import { beginRunPhase } from "../execution-preparation-timing.js";
import { logActivity } from "../activity-log.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { nativeSha256 } from "./canonical.js";
import { NATIVE_WORKSPACE_OWNER_UNVERIFIED, NATIVE_WORKSPACE_OWNER_WAIT_NEXT_ACTION } from "./native-workspace-finalization-recovery-state.js";
import { randomUUID } from "node:crypto";
import os from "node:os";
import { and, eq, inArray, sql } from "drizzle-orm";
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js";
import { withWorkspaceLockKeepalive } from "./native-workspace-lock-keepalive.js";
import {
currentNativeControllerIdentity,
evaluateNativeControllerTakeover,
type NativeControllerIdentity,
} from "./native-restart-recovery.js";
import { currentWorkspaceWorker, requestWorkspaceWorkerStopProof, type WorkspaceWorker, type WorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js";
import { currentProcessGroupId, proveLocalWorkspaceOwnerStopped, type LocalWorkspaceOwnerStop } from "./native-workspace-owner-stop-proof.js";
import { currentDeploymentIdentity, proveOwnerGone } from "../owner-proof.js";
const OWNER_KEY = "nativeWorkspaceFinalizationOwner";
export function nativeWorkspaceFinalizationLockKey(companyId: string, runId: string): string {
return `native-workspace-finalization:${companyId}:${runId}`;
}
export async function tryLockNativeWorkspaceFinalization(tx: Db, input: { companyId: string; runId: string }): Promise<boolean> {
const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(
${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>;
return lock.acquired;
}
const activeOwnerTokens = new Set<string>();
const activeRunOwners = new Map<string, { token: string; db: Db }>();
// Positive same-boot evidence, retained only when durable cleanup could not be
// confirmed. Absence from the active set alone never proves another token joined.
const joinedOwnerTokens = new Set<string>();
type Owner = {
token: string;
controllerBootId?: string;
hostname: string;
pid: number;
processStartedAt: string;
/** Copyback children inherit this group; a successor uses it to prove they stopped. */
processGroupId?: number;
worker?: WorkspaceWorker;
/** The platform volume this deployment mounted; it binds a later deployment proof. */
volumeId?: string;
/** Recorded on Railway receipts; false keeps a later deployment proof from applying. */
exclusiveVolume?: boolean;
};
export type NativeWorkspaceFinalizationWaitReason =
| "busy"
| "owner_active"
| "platform_stop_pending"
| "local_stop_pending"
| "owner_unverified"
| "binding_changed";
export type NativeWorkspaceFinalizationOwnership = {
token: string;
/** Recheck both the physical lock connection and the durable owner before publishing. */
assertHeld(): Promise<void>;
};
export class NativeWorkspaceFinalizationBusyError extends Error {
constructor() { super("native_workspace_finalization_busy"); }
}
export class NativeWorkspaceFinalizationOwnershipLostError extends Error {
constructor(options?: ErrorOptions) {
super("native_workspace_finalization_ownership_lost", options);
}
}
function readOwner(value: unknown): Owner | null {
if (!value || typeof value !== "object") return null;
const owner = value as Partial<Owner>;
return typeof owner.token === "string" && typeof owner.hostname === "string"
&& Number.isInteger(owner.pid) && Number(owner.pid) > 0
&& typeof owner.processStartedAt === "string" && Number.isFinite(Date.parse(owner.processStartedAt))
? owner as Owner : null;
}
export function nativeWorkspaceFinalizationOwnerCanUpgrade(profile: Record<string, unknown> | null,
identity: NativeControllerIdentity, worker: WorkspaceWorker | undefined): boolean {
const raw = profile?.[OWNER_KEY];
if (raw == null) return true;
const owner = readOwner(raw);
return Boolean(owner && owner.token.length > 0 && owner.token.length <= 200 && worker && owner.worker
&& Object.keys(owner.worker).length === Object.keys(worker).length && owner.controllerBootId === identity.bootId
&& owner.hostname === os.hostname() && owner.pid === identity.pid
&& Date.parse(owner.processStartedAt) === identity.processStartedAt.getTime()
&& Object.keys(worker).every(key => owner.worker![key as keyof WorkspaceWorker] === worker[key as keyof WorkspaceWorker]));
}
/**
* The advisory lock serializes controllers without holding row locks across I/O.
* The durable receipt additionally prevents a disconnected lock session from
* authorizing takeover while its controller is still doing physical copyback.
* There is deliberately no wall-clock timeout that can steal a slow export.
*/
export async function withNativeWorkspaceFinalizationOwnership<T>(
input: { db: Db; companyId: string; runId: string; retainedRepairActionId?: string },
action: (ownership: NativeWorkspaceFinalizationOwnership) => Promise<T>,
): Promise<{ acquired: false; reason: NativeWorkspaceFinalizationWaitReason } | { acquired: true; value: T }> {
const identity = await currentNativeControllerIdentity();
const processGroupId = await currentProcessGroupId();
const deployment = currentDeploymentIdentity();
const owner: Owner = {
token: randomUUID(), controllerBootId: identity.bootId, hostname: os.hostname(), pid: identity.pid,
processStartedAt: identity.processStartedAt.toISOString(),
...(processGroupId ? { processGroupId } : {}), worker: currentWorkspaceWorker(),
// Record a volume only when this deployment is volume-exclusive by the same rule the proof uses.
...(deployment ? { exclusiveVolume: deployment.exclusiveVolume } : {}),
...(deployment?.exclusiveVolume && deployment.volumeId ? { volumeId: deployment.volumeId } : {}),
};
const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
input.retainedRepairActionId ? inArray(heartbeatRuns.runtimeMode, ["native", "legacy"]) : eq(heartbeatRuns.runtimeMode, "native"));
const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`);
const runKey = nativeWorkspaceFinalizationLockKey(input.companyId, input.runId);
let claimed = false;
let actionPromise: Promise<T> | undefined;
const finishAcquisition = beginRunPhase("execution.workspace_ownership_wait");
try {
return await withDedicatedDbConnection(input.db, (dedicated) => dedicated.transaction(async (lock) => {
const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired,
(select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`);
if (rows[0]?.acquired !== true) return {
acquired: false,
reason: !input.retainedRepairActionId && activeRunOwners.get(runKey)?.db === input.db ? "owner_active" : "busy",
} as const;
return withWorkspaceLockKeepalive(lock as unknown as Db, Number(rows[0].idle_timeout_ms), async () => {
// Reads and durable progress use the ordinary pool, not the lock transaction.
const [run] = await input.db.select({ profile: heartbeatRuns.runnerProfileJson,
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')`,
agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(scope).limit(1);
if (!run) throw new Error("native_workspace_finalization_binding_missing");
const rawPrior = run.profile?.[OWNER_KEY];
const prior = readOwner(rawPrior);
let automaticResultId: string | undefined;
const currentRecoveryResult = async (db: Db) => {
if (!run.issueId) return null;
// Use the arbiter's lock order and release row locks before copyback I/O.
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId),
eq(nativeRunFinalizations.issueId, run.issueId),
)).for("update");
const [issue] = await db.select().from(issues).where(and(
eq(issues.companyId, input.companyId), eq(issues.id, run.issueId),
)).for("update");
const [currentRun] = await db.select().from(heartbeatRuns).where(scope).for("update");
if (input.retainedRepairActionId) {
const [repair] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId),
eq(issueRecoveryActions.id, input.retainedRepairActionId),
)).for("update");
const continuation = repair?.evidence.explicitUserContinuation as {
runId?: string; readiness?: { mode?: string; sourceRunId?: string };
} | undefined;
const independentRun = continuation?.readiness?.mode === "fresh_isolated_workspace"
&& continuation.readiness.sourceRunId === input.runId ? continuation.runId : undefined;
return currentRun && issue && repair && repair.evidence.runId === currentRun.id
&& repair.evidence.workspaceRecovery && !coordinator?.leaseOwner
&& ["failed", "interrupted", "timed_out", "cancelled", "succeeded"].includes(currentRun.status)
&& (!issue.executionRunId || issue.executionRunId === independentRun)
&& (!issue.checkoutRunId || issue.checkoutRunId === independentRun) ? repair : null;
}
return coordinator && issue && currentRun ? currentNativeWorkspaceRecoveryResult(db, {
companyId: input.companyId, coordinator, issue, run: currentRun,
}) : null;
};
let platformProof: WorkspaceWorkerStopProof | null = null;
let platformStopPending = false;
let localProof: Extract<LocalWorkspaceOwnerStop, { status: "verified" }>["proof"] | null = null;
let localStopPending = false;
let deploymentProof: { reason: "deployment_replaced"; previousDeploymentId: string; worker: WorkspaceWorker } | null = null;
const waitReason = () => platformStopPending ? "platform_stop_pending" as const
: localStopPending ? "local_stop_pending" as const : "owner_unverified" as const;
if (rawPrior != null) {
const requireStopVerification = async () => {
if (input.retainedRepairActionId) return { acquired: false, reason: "owner_unverified" } as const;
const fingerprint = nativeSha256({ runId: input.runId, owner: rawPrior });
const nextAction = NATIVE_WORKSPACE_OWNER_WAIT_NEXT_ACTION;
const recovery = issueRecoveryActionService(input.db);
const existing = run.issueId ? await recovery.getActiveForIssue(input.companyId, run.issueId) : null;
// Rechecking the same physical owner is not a new recovery attempt.
if (existing?.cause === NATIVE_WORKSPACE_OWNER_UNVERIFIED && existing.fingerprint === fingerprint) {
if (!existing.evidence?.workspaceOwnerStop) {
await input.db.update(issueRecoveryActions).set({ nextAction, updatedAt: new Date(),
evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"operator_required"}'::jsonb)`,
}).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, existing.id),
eq(issueRecoveryActions.fingerprint, fingerprint), eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
sql`(${issueRecoveryActions.nextAction} is distinct from ${nextAction} or ${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' is distinct from 'operator_required')`,
));
}
return { acquired: false, reason: waitReason() } as const;
}
if (run.issueId) await recovery.upsertSourceScoped({
companyId: input.companyId, sourceIssueId: run.issueId,
kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: run.agentId,
cause: NATIVE_WORKSPACE_OWNER_UNVERIFIED,
fingerprint,
evidence: { runId: input.runId, owner: rawPrior, workspaceOwnerRecovery: { state: "operator_required" } },
nextAction,
wakePolicy: null, maxAttempts: 1, supersedeOnIdentityChange: true,
});
return { acquired: false, reason: waitReason() } as const;
};
// The harness checks the exact owned deployment, including orphaned children.
// A live deployment, unsigned answer or older receipt stays fenced.
if (prior?.controllerBootId === identity.bootId && prior.pid === identity.pid
&& activeOwnerTokens.has(prior.token)) {
return { acquired: false, reason: input.retainedRepairActionId ? "owner_unverified"
: activeRunOwners.get(runKey)?.db === input.db ? "owner_active" : "busy" } as const;
}
if (prior?.worker) {
// A restart onto a newer deployment of a volume-backed service is
// proof by itself: the platform stopped the old deployment first.
const replaced = await proveOwnerGone({ kind: "local", scope: "native_workspace_finalization", generation: null,
host: prior.hostname, processSpace: null, bootId: prior.controllerBootId ?? null, pid: prior.pid,
processStartedAt: prior.processStartedAt, worker: prior.worker,
volumeId: typeof prior.volumeId === "string" ? prior.volumeId : null,
exclusiveVolume: typeof prior.exclusiveVolume === "boolean" ? prior.exclusiveVolume : null });
if (replaced.verdict === "gone" && replaced.reason === "deployment_replaced")
deploymentProof = { reason: "deployment_replaced", previousDeploymentId: prior.worker.deploymentId, worker: prior.worker };
if (!deploymentProof) {
const check = await requestWorkspaceWorkerStopProof({ runId: input.runId, ownerToken: prior.token, worker: prior.worker });
platformProof = check.status === "verified" ? check.proof : null;
platformStopPending = check.status === "pending";
}
}
if (!prior) return requireStopVerification();
if (!platformProof && !deploymentProof && prior.hostname !== owner.hostname) return requireStopVerification();
const knownJoinedHere = prior.controllerBootId === identity.bootId
&& prior.pid === identity.pid && joinedOwnerTokens.has(prior.token)
&& !activeOwnerTokens.has(prior.token);
if (!platformProof && !deploymentProof && !knownJoinedHere) {
const takeover = await evaluateNativeControllerTakeover({
owner: { leaseOwner: prior.token, leaseExpiresAt: new Date(0), controllerPid: prior.pid,
controllerProcessStartedAt: new Date(prior.processStartedAt) }, now: new Date(),
});
if (!takeover.allowed) return requireStopVerification();
// A dead parent may have orphaned tar/Git children. Only a durable
// completed-copyback barrier proves they joined; otherwise board proof
// is required even on the same host. Both callers reuse this barrier.
const [completedCopyback] = await input.db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId),
eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.issueId, run.issueId!),
eq(workspaceOperations.phase, "workspace_finalize"),
eq(workspaceOperations.status, "succeeded"),
)).limit(1);
if (!completedCopyback) {
// A restart can kill the controller mid-copyback. It can never
// confirm its own stop, so prove on this host that it and every
// copyback child in its process group exited (RC-47).
const local = await proveLocalWorkspaceOwnerStopped({ owner: prior, identity });
if (local.status !== "verified") {
localStopPending = local.status === "pending";
return requireStopVerification();
}
localProof = local.proof;
}
}
}
// Claim before the potentially ambiguous write so finally can remove
// only our token even if the server committed but its response was lost.
const claimOwner = async (db: Db) => {
activeOwnerTokens.add(owner.token);
activeRunOwners.set(runKey, { token: owner.token, db: input.db });
claimed = true;
const [claimedRun] = await db.update(heartbeatRuns).set({
runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb) || ${JSON.stringify(platformProof ? { nativeWorkspaceOwnerStopProof: platformProof } : {})}::jsonb`,
}).where(and(scope, prior
? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY} = ${JSON.stringify(rawPrior)}::jsonb`
: sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null`
)).returning({ id: heartbeatRuns.id });
return claimedRun;
};
// A stop proof (platform or local) shows the old worker exited, not that its
// task is still current. Revalidate after the proof and atomically with the CAS.
const claimedRun = platformProof || localProof || deploymentProof ? await input.db.transaction(async tx => {
const result = await currentRecoveryResult(tx as unknown as Db);
if (!result) return null;
const claimedRun = await claimOwner(tx as unknown as Db);
if (claimedRun) automaticResultId = result.id;
return claimedRun;
}) : await claimOwner(input.db);
if (!claimedRun) {
activeOwnerTokens.delete(owner.token);
if (activeRunOwners.get(runKey)?.token === owner.token) activeRunOwners.delete(runKey);
claimed = false;
return { acquired: false, reason: "binding_changed" } as const;
}
if ((platformProof || localProof || deploymentProof) && run.issueId) await logActivity(input.db, {
companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
action: "issue.workspace_owner_stop_verified", entityType: "issue", entityId: run.issueId,
runId: input.runId, details: platformProof ? { proof: platformProof } : deploymentProof ? { deploymentProof } : { localProof },
});
if (prior) joinedOwnerTokens.delete(prior.token);
const ownership: NativeWorkspaceFinalizationOwnership = {
token: owner.token,
assertHeld: async () => {
try {
// A failed transaction is never transparently reconnected by the DB client.
await lock.execute(sql`select 1`);
const [current] = await input.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(owned).limit(1);
if (!current) throw new Error("owner_replaced");
if (automaticResultId) {
const result = await input.db.transaction(tx => currentRecoveryResult(tx as unknown as Db));
if (result?.id !== automaticResultId) throw new Error("task_replaced");
}
} catch (cause) {
throw new NativeWorkspaceFinalizationOwnershipLostError({ cause });
}
},
};
await ownership.assertHeld();
if ((platformProof || localProof || deploymentProof) && run.issueId) await input.db.update(issueRecoveryActions).set({
evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"resuming"}'::jsonb)`,
updatedAt: new Date(),
}).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId),
eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED),
eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
));
finishAcquisition();
actionPromise = action(ownership);
const value = await actionPromise;
await ownership.assertHeld();
return { acquired: true, value } as const;
});
}));
} catch (error) {
finishAcquisition(true);
throw error;
} finally {
finishAcquisition();
// postgres.js may reject the transaction when the socket dies before the
// callback settles. Join that callback before releasing its durable fence.
await actionPromise?.catch(() => undefined);
if (claimed) {
activeOwnerTokens.delete(owner.token);
if (activeRunOwners.get(runKey)?.token === owner.token) activeRunOwners.delete(runKey);
joinedOwnerTokens.add(owner.token);
try {
await input.db.transaction(async tx => {
await notifyNativeResultCommit(tx, { companyId: input.companyId, runId: input.runId });
await notifyNativeWorkspaceFinalization(tx, { companyId: input.companyId, runId: input.runId });
await notifyNativeStatusReassessment(tx, { companyId: input.companyId, runId: input.runId });
await notifyQueuedRunDispatch(tx, { companyId: input.companyId });
// Saved messages also wait on this owner after the run finished.
await notifyDeferredExecutionWork(tx, { companyId: input.companyId });
await tx.update(heartbeatRuns).set({
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`,
}).where(owned);
});
joinedOwnerTokens.delete(owner.token);
} catch {
// A reconnect in this exact controller boot can use the positive join
// receipt. A new boot still needs durable barrier or operator proof.
} finally {
releaseNativeWorkspaceFinalization(input.db, { companyId: input.companyId, runId: input.runId });
}
}
}
}