fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time

## Linked Issues or Issue Description

**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.

**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.

**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.

**Paperclip version or commit**
`master` at 2b69cdfcd6.

## What Changed

- `native-safe-replacement.ts`: an unverified-ownership decision now
waits only until 5 minutes after the run's terminal time (`finished_at`,
else the finalization's last update), compared with the database
`now()`. After that, the pass records `replacementDenied` and the
existing watchdog hold, with a plain-language next action. The expiry is
checked again inside the locked transaction, so two servers cannot both
act, and a successor that a concurrent pass committed first still wins.
- The deadline applies only to `native_provider_terminal_failed` runs
whose finalization has no lease owner. Other failure codes and
person-owned causes are unchanged.
- The bound reuses `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) from
`recovery-continuation-pacing.ts`, as `NATIVE_OWNERSHIP_PROOF_BOUND_MS`.
- The wake has no time window. The wake query returns the deadline of
every marked run that is still undecided (no successor, no denial, no
lease owner), with the same eligibility as the denial. When that
deadline is past, the queue asks for a pass at once. That pass always
removes the run from the query:
  - It denies the run, which ends the wait.
  - Or it starts the exact replacement, when ownership was proven.
- Or it finds the run legitimately undecidable and clears the marker.
This happens when the history has unresolved interactions, or when the
task is not eligible for a successor (for example in backlog). The
existing task and interaction notifications wake the queue again later.
- More than 25 overdue runs: the scan cursor requests the next pass at
once.
- A lost lock or any other error: the pass throws, and the coordinator
retries after its existing `retryMs`
(`EXECUTION_RECONCILIATION_INTERVAL_MS`, 15 seconds). That is the same
backoff as every replacement-decision error today.
- An environment lease that is never released no longer blocks the
decision after the deadline. Before the deadline, the run waits as
before and records the marker. After it, an unreleased lease counts as
"provider not proven stopped", so the pass denies the run and never
starts an exact replacement next to a live lease. The fallback then
treats the unreleased lease as pending cleanup and picks
`fresh_isolated_workspace`. Admission (`heartbeat/queue.ts`, the
wake-queue adapter) does not block a run that a fresh continuation
superseded, so the new run does not wait for or touch the old lease. The
environment orphan-settlement modules are not changed.
- When the pass first sees unverified ownership before the deadline, it
writes `ownershipUnverifiedAt` once into the failure detail. It does not
change `updated_at`. This marker only limits which runs schedule a wake.
The deadline still counts from the terminal time.
- `execution-control-work.ts`: when no replacement is pending, the
replacement queue schedules its next pass at the earliest future
deadline of a marked run. This is a single timer, not a polling loop.
Rows already past their deadline are handled by the pass that runs at
startup, with or without the marker.
- No changes to `execution-recovery-resolution.ts` or
`fresh-continuation-fallback.ts`. The fallback already selects
`fresh_isolated_workspace` when its own owner proof cannot show that the
old local writer ended, `remote_epoch` for a remote workspace, and waits
(bounded) only for a process proven alive on this host.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-safe-replacement.test.ts`: new cases:
- Past the deadline (6 minutes, and 2 days for a row stuck before this
release): denial recorded on the first pass, no exact successor, then
exactly one fresh continuation with `fresh_isolated_workspace` after two
resolver passes.
- Before the deadline: no denial, no hold, no continuation, and the
replacement schedule wakes at terminal time + 5 minutes.
- Ownership proven before the deadline: the exact replacement is
scheduled and no denial is written.
- Red before the fix: 3 of the 4 new cases failed (no denial written; no
wake scheduled). The exact-replacement case passed, as expected, because
it guards existing behavior. Green after: 93 of 93 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `tsc --noEmit` for `server`: clean.
- Review fixes, red before and green after:
- A deadline that passed 10 minutes before the schedule was asked: the
queue requests an immediate pass, which denies the run and continues it
isolated.
- An overdue run with an unreleased environment lease: denied, no exact
successor, continued with `fresh_isolated_workspace`.
- An overdue run the pass cannot decide (task in backlog): one wake,
then no second wake.
- 26 overdue runs: 25 decided on the first pass, all 26 on the second,
then no wake. This one also passed before, because the existing scan
cursor covers it. It stays as a guard.
  - On the earlier head, the first three failed.
- Green after all commits: 97 of 97 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `pnpm verify` selected 2624 of 3444 test files, because
`execution-control-work.ts` is widely imported. I stopped it on a
heavily loaded machine (load average about 20). Its partial output
showed only unrelated attention-test timeouts near 30 seconds. Required
CI supplies the broad coverage.

## Risks

- After 5 minutes, a run whose old provider is truly still alive on
another server is continued in a new isolated workspace. The old writer
cannot write to it, and its files are not applied. A process proven
alive on this server still makes the fallback wait for its own bound
first.
- Deploy burst: rows stuck before this release are denied on the first
pass, at most 25 per pass (existing scan limit). Continuations then go
through the existing pacing: 6 per task per hour, 5 per company per
pass, one per agent per pass, and the agent's concurrency limit.
- No schema or API change. No change before `turn.start` for healthy
runs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-13 06:03:53 -06:00
1 parent 3fc2198759
commit 18fb6ee571
3 files changed
+178 -10

No files matched your search

@@ -6,7 +6,7 @@ import { type DeliveryQueue } from "./delivery-work-notifications.js";
import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js";
import { heldExecutionsAwaitingFreshContinuation, nextFreshContinuationCheckAt } from "./fresh-continuation-fallback.js";
import { hasPendingNativeReplacements } from "./native-runtime/native-safe-replacement.js";
import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from "./native-runtime/native-safe-replacement.js";
import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js";
import { hasPendingPauseResume } from "./issue-tree-resume.js";
import { hasPendingBudgetResume } from "./budget-resume-continuation.js";
@@ -89,7 +89,9 @@ export function executionControlSchedule(db: Db) {
return at === null ? null : deadline([new Date(at)]);
},
async replacement() {
return await hasPendingNativeReplacements(db) ? retry() : null;
if (await hasPendingNativeReplacements(db)) return retry();
const at = await nextNativeOwnershipProofDeadlineAt(db);
return at === null ? null : deadline([new Date(at)]);
},
async continuation() {
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
@@ -31,6 +31,7 @@ import {
agents,
companies,
createDb,
environmentLeases,
subscribeDatabaseWork,
heartbeatRunEvents,
heartbeatRuns,
@@ -458,6 +459,114 @@ const support = externalDatabaseUrl
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
expect(action.status).toBe("active");
});
async function seedUnprovenOwner(finishedAgoMs: number) {
const source = await seed();
await db.update(heartbeatRuns).set({ finishedAt: sql`now() - ${finishedAgoMs}::integer * interval '1 millisecond'`,
processPid: process.pid, processHostHash: "another-server" }).where(eq(heartbeatRuns.id, source.runId));
invalidateNativeReplacementDiscovery(db);
return source;
}
async function freshContinuations(issueId: string) {
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
return holds.flatMap(hold => hold.evidence.freshContinuation ? [hold.evidence.freshContinuation as Record<string, unknown>] : []);
}
it.each([["just past the deadline", 6 * 60_000], ["stuck since before this release", 2 * 24 * 60 * 60_000]] as const)(
"continues a terminal failure whose previous provider is never proven stopped (%s)", async (_label, finishedAgoMs) => {
const source = await seedUnprovenOwner(finishedAgoMs);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId));
expect(decision!.failureDetail).toMatchObject({ replacementDenied: "provider_ownership_unverified" });
expect(String(decision!.failureDetail?.nextAction)).not.toMatch(/[a-z]+_[a-z_]+|[0-9a-f]{8}-/);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(0);
expect(await executionControlSchedule(db).replacement()).toBeNull();
await settleUnrecoverableExecutions(db);
await settleUnrecoverableExecutions(db);
const continued = await freshContinuations(source.issueId);
expect(continued).toHaveLength(1);
expect(continued[0]).toMatchObject({ sourceRunId: source.runId, isolation: "fresh_isolated_workspace" });
const [task] = await db.select().from(issues).where(eq(issues.id, source.issueId));
expect(task!.status).not.toBe("blocked");
});
it("keeps waiting for ownership proof before the deadline and wakes at it", async () => {
const source = await seedUnprovenOwner(60_000);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId));
expect(decision!.failureDetail?.replacementDenied).toBeUndefined();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.runId));
const wakeAt = await executionControlSchedule(db).replacement();
expect(wakeAt).not.toBeNull();
expect(Math.abs(wakeAt! - (run!.finishedAt!.getTime() + 5 * 60_000))).toBeLessThan(2_000);
await settleUnrecoverableExecutions(db);
expect(await freshContinuations(source.issueId)).toHaveLength(0);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId))).toHaveLength(0);
});
async function denial(runId: string) {
const [row] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
return row!.failureDetail?.replacementDenied;
}
it("requests a pass for a deadline that passed long before anyone asked", async () => {
const source = await seedUnprovenOwner(60_000);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '15 minutes'` })
.where(eq(heartbeatRuns.id, source.runId));
const wakeAt = await executionControlSchedule(db).replacement();
expect(wakeAt).not.toBeNull();
expect(wakeAt!).toBeLessThanOrEqual(Date.now() + 20_000);
await reconcileSafeNativeReplacements(db);
expect(await denial(source.runId)).toBe("provider_ownership_unverified");
expect(await executionControlSchedule(db).replacement()).toBeNull();
await settleUnrecoverableExecutions(db);
expect(await freshContinuations(source.issueId)).toMatchObject([{ isolation: "fresh_isolated_workspace" }]);
});
it("stops waiting for an environment teardown that never finishes", async () => {
const source = await seed();
await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '1 minute'` }).where(eq(heartbeatRuns.id, source.runId));
await db.insert(environmentLeases).values({ companyId: source.companyId, issueId: source.issueId,
heartbeatRunId: source.runId, status: "active" });
invalidateNativeReplacementDiscovery(db);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
expect(await denial(source.runId)).toBeUndefined();
expect(await executionControlSchedule(db).replacement()).not.toBeNull();
await db.update(heartbeatRuns).set({ finishedAt: sql`now() - interval '10 minutes'` }).where(eq(heartbeatRuns.id, source.runId));
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(0);
expect(await denial(source.runId)).toBe("provider_ownership_unverified");
await settleUnrecoverableExecutions(db);
expect(await freshContinuations(source.issueId)).toMatchObject([{ sourceRunId: source.runId, isolation: "fresh_isolated_workspace" }]);
});
it("does not keep waking for an overdue run the pass cannot decide", async () => {
const source = await seedUnprovenOwner(60_000);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
await db.update(heartbeatRuns).set({ processPid: null, processHostHash: null, finishedAt: sql`now() - interval '15 minutes'` })
.where(eq(heartbeatRuns.id, source.runId));
await issueService(db).update(source.issueId, { status: "backlog" });
expect(await executionControlSchedule(db).replacement()).not.toBeNull();
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
expect(await denial(source.runId)).toBeUndefined();
expect(await executionControlSchedule(db).replacement()).toBeNull();
});
it("decides every overdue run across successive passes", async () => {
const sources: Awaited<ReturnType<typeof seed>>[] = [];
for (let index = 0; index < 26; index++) sources.push(await seedUnprovenOwner(10 * 60_000));
await reconcileSafeNativeReplacements(db);
const decided = async () => (await Promise.all(sources.map(source => denial(source.runId)))).filter(Boolean).length;
expect(await decided()).toBe(25);
expect(await executionControlSchedule(db).replacement()).not.toBeNull();
await reconcileSafeNativeReplacements(db);
expect(await decided()).toBe(26);
expect(await executionControlSchedule(db).replacement()).toBeNull();
}, 120_000);
it("still starts the exact replacement when ownership is proven before the deadline", async () => {
const source = await seedUnprovenOwner(60_000);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
await db.update(heartbeatRuns).set({ processPid: null, processHostHash: null }).where(eq(heartbeatRuns.id, source.runId));
invalidateNativeReplacementDiscovery(db);
expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 });
const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId));
expect(decision!.failureDetail?.replacementDenied).toBeUndefined();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1);
expect(await executionControlSchedule(db).replacement()).toBeNull();
});
it.each(["closed", "reassigned", "new_execution"])("closes stale recovery after %s without changing task state or granting replay", async change => {
const source = await seed(3);
await reconcileSafeNativeReplacements(db);
@@ -31,6 +31,7 @@ import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-sou
import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js";
import { nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { verifiedWorkspaceSnapshotReuse } from "../remote-workspace-snapshot-reuse-receipt.js";
import { RECOVERY_CONTINUATION_MAX_HOLD_MS } from "../recovery-continuation-pacing.js";
export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement";
const record = (v: unknown): Record<string, unknown> =>
@@ -249,6 +250,45 @@ export function nativeReplacementPending() {
);
}
export const NATIVE_OWNERSHIP_PROOF_BOUND_MS = RECOVERY_CONTINUATION_MAX_HOLD_MS;
const OWNERSHIP_UNVERIFIED = "provider_ownership_unverified";
const ownershipProofDeadline = () => sql`(coalesce(${heartbeatRuns.finishedAt}, ${nativeRunFinalizations.updatedAt})
+ ${NATIVE_OWNERSHIP_PROOF_BOUND_MS}::integer * interval '1 millisecond')`;
const ownershipProofWaiting = () => and(
eq(nativeRunFinalizations.failureCode, "native_provider_terminal_failed"), isNull(nativeRunFinalizations.leaseOwner));
async function ownershipProofExpired(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
const [row] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), nativeReplacementPending(),
ownershipProofWaiting(), sql`${ownershipProofDeadline()} <= now()`,
)).limit(1);
return Boolean(row);
}
async function markOwnershipProofWaiting(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
await db.update(nativeRunFinalizations).set({
failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb)
|| jsonb_build_object('ownershipUnverifiedAt', now())`,
}).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
ownershipProofWaiting(), sql`not (coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) ? 'ownershipUnverifiedAt')`));
}
async function releaseOwnershipProofWake(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
await db.update(nativeRunFinalizations).set({
failureDetail: sql`${nativeRunFinalizations.failureDetail} - 'ownershipUnverifiedAt'`,
}).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`));
}
export async function nextNativeOwnershipProofDeadlineAt(db: Db): Promise<number | null> {
const [row] = await db.select({ at: sql<number | null>`(extract(epoch from min(${ownershipProofDeadline()})) * 1000)::float8` })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(nativeReplacementPending(), ownershipProofWaiting(),
sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`));
return row?.at == null ? null : Math.ceil(Number(row.at));
}
const REPLACEMENT_SCAN_LIMIT = 100;
const replacementDiscovery = new WeakMap<object, { after: string | undefined; complete: boolean; generation: number; rescan: boolean }>();
function replacementDiscoveryState(db: Db) {
@@ -383,8 +423,14 @@ export async function reconcileSafeNativeReplacements(
eq(environmentLeases.heartbeatRunId, run.id),
),
);
let overdue: boolean | undefined;
const ownershipOverdue = async () => overdue ??= await ownershipProofExpired(db, run);
const leaseHeld = leases.some((lease) => lease.releasedAt === null);
// Teardown is still in progress. The next sweep rechecks its durable outcome.
if (leases.some((lease) => lease.releasedAt === null)) continue;
if (leaseHeld && !await ownershipOverdue()) {
if (coordinator.failureCode === "native_provider_terminal_failed") await markOwnershipProofWaiting(db, run);
continue;
}
const historicalListener = coordinator.failureCode === "native_session_interrupted";
if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE;
@@ -483,7 +529,10 @@ export async function reconcileSafeNativeReplacements(
: [];
});
const history = await replacementHistoryReady(db, run, coordinator.issueId);
if (history === false) continue;
if (history === false) {
if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run);
continue;
}
const historyComplete = history === true;
const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases);
if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue;
@@ -508,7 +557,7 @@ export async function reconcileSafeNativeReplacements(
providerStopped:
!processAlive(run.processPid) &&
!processAlive(run.processGroupId ? -run.processGroupId : null) &&
(remoteWorkspace?.providerStopped ?? true),
(remoteWorkspace?.providerStopped ?? true) && !leaseHeld,
workspacePreserved: noWorkspaceToSave || (remoteWorkspace ? remoteWorkspace.workspacePreserved :
typeof workspace.cwd === "string" &&
(await stat(workspace.cwd).then(
@@ -526,7 +575,14 @@ export async function reconcileSafeNativeReplacements(
});
if (!decision.allowed) {
// Lack of containment can be temporary; do not prevent the next proof.
if (decision.cause === "provider_ownership_unverified") continue;
const ownershipBound = decision.cause === OWNERSHIP_UNVERIFIED;
if (ownershipBound && !await ownershipOverdue()) {
await markOwnershipProofWaiting(db, run);
continue;
}
const nextAction = ownershipBound
? "The previous session could not be confirmed stopped, so the task continues in a fresh run with its own workspace."
: decision.nextAction;
await db.transaction(async (tx) => {
await tx.execute(
sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`,
@@ -550,7 +606,8 @@ export async function reconcileSafeNativeReplacements(
!current ||
current.phase !== "terminal_failure" ||
current.failureDetail?.successorRunId ||
current.failureDetail?.replacementDenied
current.failureDetail?.replacementDenied ||
(ownershipBound && !await ownershipProofExpired(tx, run))
)
return;
await notifyExecutionRunWork(tx, { companyId: run.companyId });
@@ -560,7 +617,7 @@ export async function reconcileSafeNativeReplacements(
failureDetail: {
...current.failureDetail,
replacementDenied: decision.cause,
nextAction: decision.nextAction,
nextAction,
},
updatedAt: now,
})
@@ -583,7 +640,7 @@ export async function reconcileSafeNativeReplacements(
cause: decision.cause,
fingerprint: `native-replacement:${run.id}`,
evidence: { runId: run.id, attempts: coordinator.attempt },
nextAction: decision.nextAction,
nextAction,
wakePolicy: null,
maxAttempts: 3,
supersedeOnIdentityChange: true,
@@ -818,7 +875,7 @@ export async function reconcileSafeNativeReplacements(
if (created) {
options.failpoint?.("lineage_committed");
scheduled += 1;
}
} else if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run);
} catch (error) {
if (options.failpoint) throw error;
pendingFailures++;