From 18fb6ee5713c0151cae3d168057a6ea0f7e1b30a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:03:53 -0600 Subject: [PATCH] fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs that end in a terminal provider failure go through automatic recovery: an exact replacement first, then a fresh continuation from saved work > - The exact replacement waits for proof that the previous provider stopped. Without that proof it writes neither a successor nor a denial > - That wait has no end. If the old sandbox or Runner can never be proven stopped (for example after a server swap), the recovery resolver keeps waiting for the replacement, and the task sits with no run and no reason a person can see > - This pull request bounds the wait. After 5 minutes on the database clock from the run's terminal time, the replacement pass records a denial, and the existing fallback continues the task in a fresh run in an isolated workspace > - The benefit is that these tasks continue by themselves instead of stalling forever, and exact replacement still wins when proof arrives in time ## Linked Issues or Issue Description **What happened?** A native run that failed with a terminal provider failure stayed stuck when its previous provider could not be proven stopped. The replacement pass skipped the run on every pass. It recorded no successor and no denial, and it set no deadline. The recovery resolver waits for one of those two records, so it never continued the task. The replacement queue also did not wake again on its own, because a completed scan reports nothing pending. **Expected behavior** Exact replacement gets a bounded first chance. After the bound, the task continues in a fresh run that cannot use the old writer's workspace. **Steps to reproduce** 1. Create a native run in `terminal_failure` with code `native_provider_terminal_failed` and a finished time more than 5 minutes ago. 2. Record a process on the run that this server cannot prove stopped (for example a PID recorded by another server). 3. Run the replacement pass and then the recovery resolver. 4. Before this change: nothing happens, on every pass. After this change: the pass records a denial and the resolver records one fresh continuation with isolation `fresh_isolated_workspace`. **Paperclip version or commit** `master` at 2b69cdfcd6. ## What Changed - `native-safe-replacement.ts`: an unverified-ownership decision now waits only until 5 minutes after the run's terminal time (`finished_at`, else the finalization's last update), compared with the database `now()`. After that, the pass records `replacementDenied` and the existing watchdog hold, with a plain-language next action. The expiry is checked again inside the locked transaction, so two servers cannot both act, and a successor that a concurrent pass committed first still wins. - The deadline applies only to `native_provider_terminal_failed` runs whose finalization has no lease owner. Other failure codes and person-owned causes are unchanged. - The bound reuses `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) from `recovery-continuation-pacing.ts`, as `NATIVE_OWNERSHIP_PROOF_BOUND_MS`. - The wake has no time window. The wake query returns the deadline of every marked run that is still undecided (no successor, no denial, no lease owner), with the same eligibility as the denial. When that deadline is past, the queue asks for a pass at once. That pass always removes the run from the query: - It denies the run, which ends the wait. - Or it starts the exact replacement, when ownership was proven. - Or it finds the run legitimately undecidable and clears the marker. This happens when the history has unresolved interactions, or when the task is not eligible for a successor (for example in backlog). The existing task and interaction notifications wake the queue again later. - More than 25 overdue runs: the scan cursor requests the next pass at once. - A lost lock or any other error: the pass throws, and the coordinator retries after its existing `retryMs` (`EXECUTION_RECONCILIATION_INTERVAL_MS`, 15 seconds). That is the same backoff as every replacement-decision error today. - An environment lease that is never released no longer blocks the decision after the deadline. Before the deadline, the run waits as before and records the marker. After it, an unreleased lease counts as "provider not proven stopped", so the pass denies the run and never starts an exact replacement next to a live lease. The fallback then treats the unreleased lease as pending cleanup and picks `fresh_isolated_workspace`. Admission (`heartbeat/queue.ts`, the wake-queue adapter) does not block a run that a fresh continuation superseded, so the new run does not wait for or touch the old lease. The environment orphan-settlement modules are not changed. - When the pass first sees unverified ownership before the deadline, it writes `ownershipUnverifiedAt` once into the failure detail. It does not change `updated_at`. This marker only limits which runs schedule a wake. The deadline still counts from the terminal time. - `execution-control-work.ts`: when no replacement is pending, the replacement queue schedules its next pass at the earliest future deadline of a marked run. This is a single timer, not a polling loop. Rows already past their deadline are handled by the pass that runs at startup, with or without the marker. - No changes to `execution-recovery-resolution.ts` or `fresh-continuation-fallback.ts`. The fallback already selects `fresh_isolated_workspace` when its own owner proof cannot show that the old local writer ended, `remote_epoch` for a remote workspace, and waits (bounded) only for a process proven alive on this host. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-safe-replacement.test.ts`: new cases: - Past the deadline (6 minutes, and 2 days for a row stuck before this release): denial recorded on the first pass, no exact successor, then exactly one fresh continuation with `fresh_isolated_workspace` after two resolver passes. - Before the deadline: no denial, no hold, no continuation, and the replacement schedule wakes at terminal time + 5 minutes. - Ownership proven before the deadline: the exact replacement is scheduled and no denial is written. - Red before the fix: 3 of the 4 new cases failed (no denial written; no wake scheduled). The exact-replacement case passed, as expected, because it guards existing behavior. Green after: 93 of 93 passed in `native-safe-replacement.test.ts` and `execution-control-work.test.ts`. - `tsc --noEmit` for `server`: clean. - Review fixes, red before and green after: - A deadline that passed 10 minutes before the schedule was asked: the queue requests an immediate pass, which denies the run and continues it isolated. - An overdue run with an unreleased environment lease: denied, no exact successor, continued with `fresh_isolated_workspace`. - An overdue run the pass cannot decide (task in backlog): one wake, then no second wake. - 26 overdue runs: 25 decided on the first pass, all 26 on the second, then no wake. This one also passed before, because the existing scan cursor covers it. It stays as a guard. - On the earlier head, the first three failed. - Green after all commits: 97 of 97 passed in `native-safe-replacement.test.ts` and `execution-control-work.test.ts`. - `pnpm verify` selected 2624 of 3444 test files, because `execution-control-work.ts` is widely imported. I stopped it on a heavily loaded machine (load average about 20). Its partial output showed only unrelated attention-test timeouts near 30 seconds. Required CI supplies the broad coverage. ## Risks - After 5 minutes, a run whose old provider is truly still alive on another server is continued in a new isolated workspace. The old writer cannot write to it, and its files are not applied. A process proven alive on this server still makes the fallback wait for its own bound first. - Deploy burst: rows stuck before this release are denied on the first pass, at most 25 per pass (existing scan limit). Continuations then go through the existing pacing: 6 per task per hour, 5 per company per pass, one per agent per pass, and the agent's concurrency limit. - No schema or API change. No change before `turn.start` for healthy runs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- server/src/services/execution-control-work.ts | 6 +- .../native-safe-replacement.test.ts | 109 ++++++++++++++++++ .../native-runtime/native-safe-replacement.ts | 73 ++++++++++-- 3 files changed, 178 insertions(+), 10 deletions(-) diff --git a/server/src/services/execution-control-work.ts b/server/src/services/execution-control-work.ts index 69981ba1647..8bc16172fb9 100644 --- a/server/src/services/execution-control-work.ts +++ b/server/src/services/execution-control-work.ts @@ -6,7 +6,7 @@ import { type DeliveryQueue } from "./delivery-work-notifications.js"; import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js"; import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js"; import { heldExecutionsAwaitingFreshContinuation, nextFreshContinuationCheckAt } from "./fresh-continuation-fallback.js"; -import { hasPendingNativeReplacements } from "./native-runtime/native-safe-replacement.js"; +import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from "./native-runtime/native-safe-replacement.js"; import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js"; import { hasPendingPauseResume } from "./issue-tree-resume.js"; import { hasPendingBudgetResume } from "./budget-resume-continuation.js"; @@ -89,7 +89,9 @@ export function executionControlSchedule(db: Db) { return at === null ? null : deadline([new Date(at)]); }, async replacement() { - return await hasPendingNativeReplacements(db) ? retry() : null; + if (await hasPendingNativeReplacements(db)) return retry(); + const at = await nextNativeOwnershipProofDeadlineAt(db); + return at === null ? null : deadline([new Date(at)]); }, async continuation() { const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( diff --git a/server/src/services/native-runtime/native-safe-replacement.test.ts b/server/src/services/native-runtime/native-safe-replacement.test.ts index 088ae1bffc6..a0fdc9116c7 100644 --- a/server/src/services/native-runtime/native-safe-replacement.test.ts +++ b/server/src/services/native-runtime/native-safe-replacement.test.ts @@ -31,6 +31,7 @@ import { agents, companies, createDb, + environmentLeases, subscribeDatabaseWork, heartbeatRunEvents, heartbeatRuns, @@ -458,6 +459,114 @@ const support = externalDatabaseUrl const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); expect(action.status).toBe("active"); }); + async function seedUnprovenOwner(finishedAgoMs: number) { + const source = await seed(); + await db.update(heartbeatRuns).set({ finishedAt: sql`now() - ${finishedAgoMs}::integer * interval '1 millisecond'`, + processPid: process.pid, processHostHash: "another-server" }).where(eq(heartbeatRuns.id, source.runId)); + invalidateNativeReplacementDiscovery(db); + return source; + } + async function freshContinuations(issueId: string) { + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + return holds.flatMap(hold => hold.evidence.freshContinuation ? [hold.evidence.freshContinuation as Record] : []); + } + it.each([["just past the deadline", 6 * 60_000], ["stuck since before this release", 2 * 24 * 60 * 60_000]] as const)( + "continues a terminal failure whose previous provider is never proven stopped (%s)", async (_label, finishedAgoMs) => { + const source = await seedUnprovenOwner(finishedAgoMs); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId)); + expect(decision!.failureDetail).toMatchObject({ replacementDenied: "provider_ownership_unverified" }); + expect(String(decision!.failureDetail?.nextAction)).not.toMatch(/[a-z]+_[a-z_]+|[0-9a-f]{8}-/); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(0); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + await settleUnrecoverableExecutions(db); + await settleUnrecoverableExecutions(db); + const continued = await freshContinuations(source.issueId); + expect(continued).toHaveLength(1); + expect(continued[0]).toMatchObject({ sourceRunId: source.runId, isolation: "fresh_isolated_workspace" }); + const [task] = await db.select().from(issues).where(eq(issues.id, source.issueId)); + expect(task!.status).not.toBe("blocked"); + }); + it("keeps waiting for ownership proof before the deadline and wakes at it", async () => { + const source = await seedUnprovenOwner(60_000); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId)); + expect(decision!.failureDetail?.replacementDenied).toBeUndefined(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.runId)); + const wakeAt = await executionControlSchedule(db).replacement(); + expect(wakeAt).not.toBeNull(); + expect(Math.abs(wakeAt! - (run!.finishedAt!.getTime() + 5 * 60_000))).toBeLessThan(2_000); + await settleUnrecoverableExecutions(db); + expect(await freshContinuations(source.issueId)).toHaveLength(0); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId))).toHaveLength(0); + }); + async function denial(runId: string) { + const [row] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + return row!.failureDetail?.replacementDenied; + } + it("requests a pass for a deadline that passed long before anyone asked", async () => { + const source = await seedUnprovenOwner(60_000); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '15 minutes'` }) + .where(eq(heartbeatRuns.id, source.runId)); + const wakeAt = await executionControlSchedule(db).replacement(); + expect(wakeAt).not.toBeNull(); + expect(wakeAt!).toBeLessThanOrEqual(Date.now() + 20_000); + await reconcileSafeNativeReplacements(db); + expect(await denial(source.runId)).toBe("provider_ownership_unverified"); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + await settleUnrecoverableExecutions(db); + expect(await freshContinuations(source.issueId)).toMatchObject([{ isolation: "fresh_isolated_workspace" }]); + }); + it("stops waiting for an environment teardown that never finishes", async () => { + const source = await seed(); + await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '1 minute'` }).where(eq(heartbeatRuns.id, source.runId)); + await db.insert(environmentLeases).values({ companyId: source.companyId, issueId: source.issueId, + heartbeatRunId: source.runId, status: "active" }); + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await denial(source.runId)).toBeUndefined(); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '10 minutes'` }).where(eq(heartbeatRuns.id, source.runId)); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(0); + expect(await denial(source.runId)).toBe("provider_ownership_unverified"); + await settleUnrecoverableExecutions(db); + expect(await freshContinuations(source.issueId)).toMatchObject([{ sourceRunId: source.runId, isolation: "fresh_isolated_workspace" }]); + }); + it("does not keep waking for an overdue run the pass cannot decide", async () => { + const source = await seedUnprovenOwner(60_000); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + await db.update(heartbeatRuns).set({ processPid: null, processHostHash: null, finishedAt: sql`now() - interval '15 minutes'` }) + .where(eq(heartbeatRuns.id, source.runId)); + await issueService(db).update(source.issueId, { status: "backlog" }); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await denial(source.runId)).toBeUndefined(); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + }); + it("decides every overdue run across successive passes", async () => { + const sources: Awaited>[] = []; + for (let index = 0; index < 26; index++) sources.push(await seedUnprovenOwner(10 * 60_000)); + await reconcileSafeNativeReplacements(db); + const decided = async () => (await Promise.all(sources.map(source => denial(source.runId)))).filter(Boolean).length; + expect(await decided()).toBe(25); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + await reconcileSafeNativeReplacements(db); + expect(await decided()).toBe(26); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + }, 120_000); + it("still starts the exact replacement when ownership is proven before the deadline", async () => { + const source = await seedUnprovenOwner(60_000); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + await db.update(heartbeatRuns).set({ processPid: null, processHostHash: null }).where(eq(heartbeatRuns.id, source.runId)); + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 }); + const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId)); + expect(decision!.failureDetail?.replacementDenied).toBeUndefined(); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + }); it.each(["closed", "reassigned", "new_execution"])("closes stale recovery after %s without changing task state or granting replay", async change => { const source = await seed(3); await reconcileSafeNativeReplacements(db); diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 91d99925903..a84c35446b6 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -31,6 +31,7 @@ import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-sou import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js"; import { nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js"; import { verifiedWorkspaceSnapshotReuse } from "../remote-workspace-snapshot-reuse-receipt.js"; +import { RECOVERY_CONTINUATION_MAX_HOLD_MS } from "../recovery-continuation-pacing.js"; export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement"; const record = (v: unknown): Record => @@ -249,6 +250,45 @@ export function nativeReplacementPending() { ); } +export const NATIVE_OWNERSHIP_PROOF_BOUND_MS = RECOVERY_CONTINUATION_MAX_HOLD_MS; +const OWNERSHIP_UNVERIFIED = "provider_ownership_unverified"; +const ownershipProofDeadline = () => sql`(coalesce(${heartbeatRuns.finishedAt}, ${nativeRunFinalizations.updatedAt}) + + ${NATIVE_OWNERSHIP_PROOF_BOUND_MS}::integer * interval '1 millisecond')`; +const ownershipProofWaiting = () => and( + eq(nativeRunFinalizations.failureCode, "native_provider_terminal_failed"), isNull(nativeRunFinalizations.leaseOwner)); + +async function ownershipProofExpired(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) { + const [row] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), nativeReplacementPending(), + ownershipProofWaiting(), sql`${ownershipProofDeadline()} <= now()`, + )).limit(1); + return Boolean(row); +} + +async function markOwnershipProofWaiting(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) { + await db.update(nativeRunFinalizations).set({ + failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) + || jsonb_build_object('ownershipUnverifiedAt', now())`, + }).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), + ownershipProofWaiting(), sql`not (coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) ? 'ownershipUnverifiedAt')`)); +} + +async function releaseOwnershipProofWake(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) { + await db.update(nativeRunFinalizations).set({ + failureDetail: sql`${nativeRunFinalizations.failureDetail} - 'ownershipUnverifiedAt'`, + }).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), + sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`)); +} + +export async function nextNativeOwnershipProofDeadlineAt(db: Db): Promise { + const [row] = await db.select({ at: sql`(extract(epoch from min(${ownershipProofDeadline()})) * 1000)::float8` }) + .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(nativeReplacementPending(), ownershipProofWaiting(), + sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`)); + return row?.at == null ? null : Math.ceil(Number(row.at)); +} + const REPLACEMENT_SCAN_LIMIT = 100; const replacementDiscovery = new WeakMap(); function replacementDiscoveryState(db: Db) { @@ -383,8 +423,14 @@ export async function reconcileSafeNativeReplacements( eq(environmentLeases.heartbeatRunId, run.id), ), ); + let overdue: boolean | undefined; + const ownershipOverdue = async () => overdue ??= await ownershipProofExpired(db, run); + const leaseHeld = leases.some((lease) => lease.releasedAt === null); // Teardown is still in progress. The next sweep rechecks its durable outcome. - if (leases.some((lease) => lease.releasedAt === null)) continue; + if (leaseHeld && !await ownershipOverdue()) { + if (coordinator.failureCode === "native_provider_terminal_failed") await markOwnershipProofWaiting(db, run); + continue; + } const historicalListener = coordinator.failureCode === "native_session_interrupted"; if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue; const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE; @@ -483,7 +529,10 @@ export async function reconcileSafeNativeReplacements( : []; }); const history = await replacementHistoryReady(db, run, coordinator.issueId); - if (history === false) continue; + if (history === false) { + if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run); + continue; + } const historyComplete = history === true; const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases); if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue; @@ -508,7 +557,7 @@ export async function reconcileSafeNativeReplacements( providerStopped: !processAlive(run.processPid) && !processAlive(run.processGroupId ? -run.processGroupId : null) && - (remoteWorkspace?.providerStopped ?? true), + (remoteWorkspace?.providerStopped ?? true) && !leaseHeld, workspacePreserved: noWorkspaceToSave || (remoteWorkspace ? remoteWorkspace.workspacePreserved : typeof workspace.cwd === "string" && (await stat(workspace.cwd).then( @@ -526,7 +575,14 @@ export async function reconcileSafeNativeReplacements( }); if (!decision.allowed) { // Lack of containment can be temporary; do not prevent the next proof. - if (decision.cause === "provider_ownership_unverified") continue; + const ownershipBound = decision.cause === OWNERSHIP_UNVERIFIED; + if (ownershipBound && !await ownershipOverdue()) { + await markOwnershipProofWaiting(db, run); + continue; + } + const nextAction = ownershipBound + ? "The previous session could not be confirmed stopped, so the task continues in a fresh run with its own workspace." + : decision.nextAction; await db.transaction(async (tx) => { await tx.execute( sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`, @@ -550,7 +606,8 @@ export async function reconcileSafeNativeReplacements( !current || current.phase !== "terminal_failure" || current.failureDetail?.successorRunId || - current.failureDetail?.replacementDenied + current.failureDetail?.replacementDenied || + (ownershipBound && !await ownershipProofExpired(tx, run)) ) return; await notifyExecutionRunWork(tx, { companyId: run.companyId }); @@ -560,7 +617,7 @@ export async function reconcileSafeNativeReplacements( failureDetail: { ...current.failureDetail, replacementDenied: decision.cause, - nextAction: decision.nextAction, + nextAction, }, updatedAt: now, }) @@ -583,7 +640,7 @@ export async function reconcileSafeNativeReplacements( cause: decision.cause, fingerprint: `native-replacement:${run.id}`, evidence: { runId: run.id, attempts: coordinator.attempt }, - nextAction: decision.nextAction, + nextAction, wakePolicy: null, maxAttempts: 3, supersedeOnIdentityChange: true, @@ -818,7 +875,7 @@ export async function reconcileSafeNativeReplacements( if (created) { options.failpoint?.("lineage_committed"); scheduled += 1; - } + } else if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run); } catch (error) { if (options.failpoint) throw error; pendingFailures++;