Files
PaperClipAI/server/src/services/native-runtime/native-finalization-reconciler.ts
T
DottaandClaude Opus 5.5 f0679ee0c7 fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.

## Linked Issues or Issue Description

**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.

**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.

**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.

## What Changed

- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.

## Verification

Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".

Local runs on this branch (based on master 566843820c):
- `cd server && pnpm exec vitest run <file>`, one file at a time:
  - `src/__tests__/assigned-work-sweep-round-trips.test.ts`: 2 passed
  - `src/__tests__/assigned-work-invariant.test.ts`: 31 passed
  - `src/services/heartbeat/retries.test.ts`: 30 passed
  - `src/__tests__/native-session-resumption.test.ts`: 20 passed
-
`src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
217 passed, 14 skipped
- `src/__tests__/heartbeat-native-restart-suspension.test.ts`: 115
passed
- `cd server && pnpm run typecheck` (CI's layered typecheck, including
test files): clean.

- `cd server && pnpm exec tsc --noEmit`: clean.
- Timing note: local round trips take about 0.2 ms, so local
milliseconds do not show the gain (736 ms became 303–583 ms on a loaded
machine). At staging's ~50 ms per round trip, the ~2,040 sequential
statements removed are about 100 s.
- No existing test expectation was changed.

## Risks

- After this change, a task the sweep continued in the last hour can get
fewer transient retries. This is intended: one task gets at most six
automatic attempts an hour. A person's Retry is not counted in this
ledger.
- The interruption deadline ends a native run that shows no provider
progress for 15 minutes across server swaps. The fallback then continues
it in a fresh isolated workspace, so the task is not left for a person
to fix. A run that shows progress resets the clock.
- Coordinators saved by older releases start the clock at their first
claim by this code, so an upgrade does not expire any run immediately.
- No migration. No schema change. The new history field is inside
existing JSON.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:10:28 -06:00

996 lines
45 KiB
TypeScript

import { sameRunResumeClaimEntry } from "./scheduled-resume-continuation.js";
import { appendBoundedRecoveryHistory } from "./native-restart-recovery.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { currentControllerProcessStamp } from "../controller-process-identity.js";
import { NATIVE_SESSION_RESUMPTION_ISSUE_STATUSES } from "./native-session-resumption-statuses.js";
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
import { pendingNativeRunProjectionCondition } from "./native-run-projection.js";
import { nativeStatusReassessmentSourceIsSuperseded, reassessNativeStatusFromSource } from "./native-status-reassessment.js";
import { resolveNativeReconciliationStatus } from "./native-reconciliation-status.js";
export { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js";
import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js";
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
import type { WorkspaceWorkers } from "../plugin-workspaces.js";
import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js";
import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js";
import { logger } from "../../middleware/logger.js";
import { DELIVERY_QUEUES, notifyDeliveryWork, subscribeDeliveryWork } from "../delivery-work-notifications.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import {
createOwnershipReportState,
DEFERRED_INTENT_NOT_PROMOTED,
USER_COMMENT_DROPPED_BY_STALE_RUN,
USER_COMMENT_WITHOUT_WAKE,
ownershipReportFingerprint,
recheckSettlingKinds,
runOwnershipReconciliationPass,
type OwnershipReport,
} from "../recovery/ownership-invariant-report.js";
import { deferredWorkIntentWait, promoteDeferredWorkIntents, saveUndeliveredUserComments } from "../work-intent-outbox.js";
import { notifyQueuedRunDispatch } from "../run-dispatch-notifications.js";
import { QUEUED_RUN_NOT_DISPATCHED } from "../queued-run-dispatch-liveness.js";
import { createHash, randomUUID } from "node:crypto";
import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, lt, lte, not, notInArray, or, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
heartbeatRunEvents,
heartbeatRuns,
issues,
nativeRunFinalizations,
nativeRunResults,
statusDecisionEffects,
statusDecisions,
workspaceOperations,
} from "@paperclipai/db";
import {
finalizeNativeRun,
recordNativeFinalizationFailure,
repairCommittedNativeReviewResponse,
repairCommittedNativeChatResponse,
} from "./native-run-finalizer.js";
import {
dispatchPendingNativeStatusEffects,
} from "./status-decision-committer.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { issueService } from "../issues.js";
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js";
import { attemptNativeWorkspaceFinalization, recordNativeWorkspaceFinalizationFailure } from "./native-workspace-finalizer.js";
import { dismissObsoleteNativePolicyReviews } from "./obsolete-policy-reviews.js";
import {
cleanupNativeWorkspaceSync,
readNativeWorkspaceSyncReference,
} from "./native-workspace-sync.js";
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
import { createNativeReconciliationPassTiming } from "./native-reconciliation-timing.js";
import {
isNativeRunnerOwnershipHeld,
nativeRunnerOwnershipNotHeldCondition,
} from "./native-runner-ownership.js";
import {
type NativeAuthoritativeIssueStatus,
} from "./status-arbiter.js";
import { readNativeSessionResumptionAgent, restoreNativeSessionExecution } from "./native-session-resumption-admission.js";
import { retireSupersededNativeRetry } from "./native-superseded-retry.js";
import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js";
function record(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? value as Record<string, unknown>
: {};
}
/** `attempt` is the coordinator attempt the claim saw; the executor bumps it when it takes provider ownership. */
export type NativeSessionResumeClaim = { runId: string; leaseOwner: string; attempt: number };
export { reconcileRetainedNativeSessionCleanups } from "./retained-native-session-cleanup.js";
export async function dispatchNativeSessionResumptions(input: {
db: Db;
runnerInstanceId: string;
runIds?: string[];
now?: Date;
limit?: number;
dispatch: (claim: NativeSessionResumeClaim) => void;
deferIneligible?: boolean;
onError?: (error: unknown) => void;
}) {
const claims = await claimNativeSessionResumptions(input);
for (const claim of claims) input.dispatch(claim);
return claims;
}
/**
* Claims explicit, result-less retryable failures for same-run recovery.
* Merely expired `observed` ownership stays blocked because lease expiry alone
* does not prove that the original provider owner stopped. Eligibility remains
* independent of the live feature flag and legacy scheduler.
*/
export async function claimNativeSessionResumptions(input: {
db: Db;
runnerInstanceId: string;
runIds?: string[];
now?: Date;
limit?: number;
deferIneligible?: boolean;
onError?: (error: unknown) => void;
}): Promise<NativeSessionResumeClaim[]> {
const now = input.now ?? new Date();
const candidates = await input.db
.select({ runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, issueId: nativeRunFinalizations.issueId })
.from(heartbeatRuns)
.innerJoin(
nativeRunFinalizations,
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
)
.where(
and(
eq(heartbeatRuns.runtimeMode, "native"),
legacyWorkspaceRecoveryNotMigrated(),
nativeRunnerOwnershipNotHeldCondition(),
isNull(heartbeatRuns.processPid),
isNull(heartbeatRuns.processGroupId),
isNull(nativeRunFinalizations.resultId),
eq(nativeRunFinalizations.phase, "retryable_failure"),
or(
isNull(nativeRunFinalizations.nextAttemptAt),
lte(nativeRunFinalizations.nextAttemptAt, now),
),
or(
isNull(nativeRunFinalizations.leaseOwner),
isNull(nativeRunFinalizations.leaseExpiresAt),
lte(nativeRunFinalizations.leaseExpiresAt, now),
),
...(input.runIds?.length
? [inArray(heartbeatRuns.id, input.runIds)]
: []),
),
)
.orderBy(sql`${nativeRunFinalizations.nextAttemptAt} asc nulls first`, asc(nativeRunFinalizations.runId))
.limit(input.limit ?? 25);
const claims: NativeSessionResumeClaim[] = [];
for (const candidate of candidates) {
const leaseOwner = `${input.runnerInstanceId}:resume:${randomUUID()}`;
let terminalRunToEmit: typeof heartbeatRuns.$inferSelect | null = null;
let claimFailed = false;
let claimedAttempt = 0;
let leaseLiveByDatabaseClock = false;
const claimed = await input.db.transaction(async (tx) => {
await tx.execute(sql`select set_config('statement_timeout', '10000', true), set_config('lock_timeout', '2000', true)`);
const [issue] = await tx.select().from(issues).where(and(
eq(issues.companyId, candidate.companyId), eq(issues.id, candidate.issueId),
)).for("update").limit(1);
if (!issue) return false;
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, candidate.companyId), eq(nativeRunFinalizations.runId, candidate.runId),
eq(nativeRunFinalizations.issueId, issue.id),
)).for("update").limit(1);
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.id, candidate.runId),
eq(heartbeatRuns.nativeIssueId, issue.id),
)).for("update").limit(1);
if (!run || !coordinator || run.runnerProfileJson?.legacyWorkspaceRecoveryMigration != null) return false;
if (await retireSupersededNativeRetry(tx as unknown as Db, {
run, coordinator, issueExecutionRunId: issue.executionRunId, now,
})) return false;
if (!(NATIVE_SESSION_RESUMPTION_ISSUE_STATUSES as readonly string[]).includes(issue.status)) return false;
const row = { run, coordinator };
if (
row.run.runtimeMode !== "native" ||
isNativeRunnerOwnershipHeld(row.run) ||
row.run.processPid !== null ||
row.run.processGroupId !== null ||
row.coordinator.resultId ||
row.coordinator.phase !== "retryable_failure" ||
(row.coordinator.nextAttemptAt &&
row.coordinator.nextAttemptAt > now) ||
(row.coordinator.leaseOwner &&
row.coordinator.leaseExpiresAt &&
row.coordinator.leaseExpiresAt > now) ||
!["running", "failed"].includes(row.run.status)
)
return false;
const agent = await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run, coordinator });
if (!agent) return false;
const profile = row.run.runnerProfileJson ?? {};
const persistedInput = profile.nativeExecutionInput;
const checkpoint = profile.sessionCheckpoint;
const failureDetail = record(row.coordinator.failureDetail);
const originalFailureCode =
typeof failureDetail.originalFailureCode === "string"
? failureDetail.originalFailureCode
: row.run.errorCode ?? row.coordinator.failureCode
?? "native_session_interrupted";
const providerEvent = await tx
.select({ id: heartbeatRunEvents.id })
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.runId, row.run.id),
inArray(heartbeatRunEvents.eventType, [
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"turn.started",
"provider.event",
"provider.rpc_result",
]),
),
)
.limit(1)
.then((rows) => rows[0] ?? null);
const checkpointRecord = record(checkpoint);
const checkpointHasProviderIdentity =
(typeof checkpointRecord.providerSessionId === "string"
&& checkpointRecord.providerSessionId.length > 0)
|| Object.keys(record(checkpointRecord.providerIdentity)).length > 0;
const bootstrapRetry =
!!persistedInput
&& !checkpoint
&& !providerEvent
&& failureDetail.recoveryMode === "bootstrap_retry"
&& failureDetail.providerSessionEstablished === false;
if (
!persistedInput
|| (!checkpoint && !bootstrapRetry)
|| (!!checkpoint && !checkpointHasProviderIdentity)
) {
const failureCode = originalFailureCode;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, row.run.companyId);
await notifyLiveRunRecovery(tx, { companyId: row.run.companyId, runId: row.run.id, agentId: row.run.agentId });
await notifyQueuedRunDispatch(tx, { companyId: row.run.companyId });
await tx.update(nativeRunFinalizations).set({
phase: "terminal_failure",
leaseOwner: null,
leaseExpiresAt: null,
failureCode,
failureDetail: {
...failureDetail,
originalFailureCode,
recoveryMode: "ambiguous_state",
providerSessionEstablished:
checkpointHasProviderIdentity || providerEvent !== null,
recoveryOwner: { kind: "board" },
nextAction: "Inspect the original provider failure; durable state is missing or ambiguous and opening a replacement provider session is forbidden.",
},
nextAttemptAt: null,
updatedAt: now,
}).where(eq(nativeRunFinalizations.runId, row.run.id));
await notifyExternalAgentSettlement(tx, { companyId: row.run.companyId, runId: row.run.id });
await notifyInstructionCollection(tx, { companyId: row.run.companyId, runId: row.run.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: row.run.companyId, runId: row.run.id });
await notifyUnsafeWorkspaceExportRunChange(tx, row.run);
const [updatedRun] = await tx.update(heartbeatRuns).set({
status: "failed",
finishedAt: now,
nativePhase: "terminal_failure",
nativePhaseUpdatedAt: now,
errorCode: failureCode,
error: typeof failureDetail.message === "string"
? failureDetail.message
: "Persisted native session state is ambiguous and cannot be resumed safely",
updatedAt: now,
}).where(eq(heartbeatRuns.id, row.run.id)).returning();
// Only a genuine transition into "failed" is a new terminal failure.
// A candidate that is already "failed" (the filter above admits
// both "running" and "failed") must not send a second Sentry event,
// unless that earlier write was an unreported pending retry.
terminalRunToEmit =
updatedRun && isNewTerminalRunFailure(row.run, updatedRun) ? updatedRun : null;
await issueService(tx as unknown as Db).update(
row.coordinator.issueId,
{ status: "blocked" },
tx,
);
await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({
companyId: row.run.companyId,
sourceIssueId: row.coordinator.issueId,
kind: "active_run_watchdog",
ownerType: "board",
ownerAgentId: null,
returnOwnerAgentId: row.run.agentId,
cause: failureCode,
fingerprint: createHash("sha256").update(`${row.run.id}:${failureCode}`).digest("hex"),
evidence: {
runId: row.run.id,
hasEnvelope: !!persistedInput,
hasCheckpoint: !!checkpoint,
providerEventsExist: providerEvent !== null,
originalFailureCode,
},
nextAction: "Inspect the original provider failure and explicitly resolve recovery; do not open a duplicate provider session.",
wakePolicy: null,
maxAttempts: 3,
supersedeOnIdentityChange: true,
});
return false;
}
const leaseExpiresAt = new Date(now.getTime() + 20 * 60_000);
await notifyLiveRunRecovery(tx, { companyId: row.run.companyId, runId: row.run.id, agentId: row.run.agentId });
const stamp = currentControllerProcessStamp();
const resumeClaim = sameRunResumeClaimEntry({ at: now, controllerBootId: stamp.bootId,
providerAttempt: row.coordinator.attempt, failureDetail: row.coordinator.failureDetail });
const updated = await tx.update(nativeRunFinalizations).set({
phase: "observed",
leaseOwner,
leaseExpiresAt,
controllerBootId: stamp.bootId, controllerPid: stamp.pid, controllerHostHash: stamp.hostHash,
controllerProcessStartedAt: stamp.processStartedAt ? new Date(stamp.processStartedAt) : null,
recoveryHistory: appendBoundedRecoveryHistory(resumeClaim),
failureCode: null,
failureDetail: null,
nextAttemptAt: null,
updatedAt: now,
}).where(and(
eq(nativeRunFinalizations.runId, row.run.id),
eq(nativeRunFinalizations.phase, row.coordinator.phase),
or(isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt),
sql`${nativeRunFinalizations.leaseExpiresAt} <= clock_timestamp()`),
)).returning({ runId: nativeRunFinalizations.runId }).then((rows) => rows[0] ?? null);
if (!updated) {
leaseLiveByDatabaseClock = true;
return false;
}
claimedAttempt = row.coordinator.attempt;
await tx.update(heartbeatRuns).set({
status: "running",
finishedAt: null,
error: null,
errorCode: null,
nativePhase: "observed",
nativePhaseUpdatedAt: now,
updatedAt: now,
}).where(eq(heartbeatRuns.id, row.run.id));
await restoreNativeSessionExecution(tx as unknown as Db, { issue, run, agentName: agent.name, now });
return true;
}).catch((error: unknown) => {
claimFailed = true;
input.onError?.(error);
terminalRunToEmit = null;
logger.warn({ err: error, companyId: candidate.companyId, runId: candidate.runId },
"failed to claim native session recovery candidate");
return false;
});
// Telemetry is best-effort background work; it must not delay claiming
// the remaining candidates in this loop, so fire it and do not await it.
if (terminalRunToEmit) {
void emitAgentTaskRun(input.db, terminalRunToEmit);
void reportRunFailure(input.db, terminalRunToEmit);
}
if (claimed) claims.push({ runId: candidate.runId, leaseOwner, attempt: claimedAttempt });
else if (!leaseLiveByDatabaseClock && (claimFailed || input.deferIneligible !== false)) await input.db.transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '10000', true), set_config('lock_timeout', '2000', true)`);
await notifyLiveRunRecovery(tx, { companyId: candidate.companyId, runId: candidate.runId });
await tx.update(nativeRunFinalizations).set({
nextAttemptAt: new Date(now.getTime() + 30_000),
updatedAt: now,
}).where(and(
eq(nativeRunFinalizations.companyId, candidate.companyId),
eq(nativeRunFinalizations.issueId, candidate.issueId),
eq(nativeRunFinalizations.runId, candidate.runId),
eq(nativeRunFinalizations.phase, "retryable_failure"),
isNull(nativeRunFinalizations.resultId),
or(isNull(nativeRunFinalizations.nextAttemptAt), lte(nativeRunFinalizations.nextAttemptAt, now)),
or(isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt), lte(nativeRunFinalizations.leaseExpiresAt, now)),
));
}).catch((error: unknown) => {
input.onError?.(error);
logger.warn({ err: error, companyId: candidate.companyId, runId: candidate.runId },
"failed to defer native session recovery candidate");
});
}
return claims;
}
type NativeReconciliationOptions = {
skipObsoletePolicyReviews?: boolean;
skipAutomaticCompletionReviews?: boolean;
skipUnsafeWorkspaceExports?: boolean;
skipNativePendingEffects?: boolean;
skipNativeResponsePresentation?: boolean;
skipNativeReviewResponse?: boolean;
skipNativeStatusReassessment?: boolean;
skipNativeWorkspaceFinalization?: boolean;
skipNativeResultCommit?: boolean;
skipNativeRunProjection?: boolean;
skipExternalConversationSettlement?: boolean;
skipExternalAgentSettlement?: boolean;
skipNativeRecoveryNotices?: boolean;
pluginWorkerManager?: WorkspaceWorkers;
environmentRuntime?: EnvironmentRuntimeService;
onWorkspaceSettled?: (input: {
runId: string;
companyId: string;
agentId: string;
succeeded: boolean;
}) => Promise<void>;
};
type NativeReconciliationPassTiming = ReturnType<typeof createNativeReconciliationPassTiming>;
export const NATIVE_SETTLED_COMMITTED_GRACE_MS = 60 * 60 * 1000;
export const NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS = 10;
const NATIVE_SETTLED_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out"];
let settledCommittedUpkeepCursor: string | null = null;
export function resetSettledCommittedUpkeepForTests() {
settledCommittedUpkeepCursor = null;
}
function settledCommittedFinalization(now: Date): SQL {
return and(
eq(nativeRunFinalizations.phase, "committed"),
inArray(heartbeatRuns.status, NATIVE_SETTLED_RUN_STATUSES),
isNotNull(heartbeatRuns.finishedAt),
lt(heartbeatRuns.finishedAt, new Date(now.getTime() - NATIVE_SETTLED_COMMITTED_GRACE_MS)),
sql<boolean>`not exists (
select 1 from ${statusDecisionEffects} effect
where effect.company_id = ${nativeRunFinalizations.companyId}
and effect.issue_id = ${nativeRunFinalizations.issueId}
and effect.decision_id = ${nativeRunFinalizations.decisionId}
and effect.delivery_state in ('pending', 'failed')
)`,
)!;
}
async function nextSettledCommittedUpkeep(db: Db, now: Date): Promise<string[]> {
const page = (after: string | null) => db
.select({ runId: heartbeatRuns.id })
.from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(
eq(heartbeatRuns.runtimeMode, "native"),
isNotNull(nativeRunFinalizations.resultId),
settledCommittedFinalization(now),
...(after ? [gt(heartbeatRuns.id, after)] : []),
))
.orderBy(asc(heartbeatRuns.id))
.limit(NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS);
let rows = await page(settledCommittedUpkeepCursor);
if (rows.length === 0 && settledCommittedUpkeepCursor !== null) rows = await page(null);
settledCommittedUpkeepCursor = rows.length === NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS
? rows[rows.length - 1]!.runId
: null;
return rows.map((row) => row.runId);
}
/** Recovery is keyed only by persisted mode/coordinator state, never the live flag. */
export async function reconcileNativeFinalizations(
db: Db,
runIds?: string[],
options: NativeReconciliationOptions = {},
) {
const timing = createNativeReconciliationPassTiming();
let outcome: "ok" | "failed" = "ok";
try {
return await reconcileNativeFinalizationsPass(db, runIds, options, timing);
} catch (error) {
outcome = "failed";
throw error;
} finally {
timing.finish({ outcome, scoped: Boolean(runIds?.length) });
}
}
async function reconcileNativeFinalizationsPass(
db: Db,
runIds: string[] | undefined,
options: NativeReconciliationOptions,
timing: NativeReconciliationPassTiming,
) {
if (!options.skipUnsafeWorkspaceExports) {
timing.phase("legacy_workspace_exports");
await recoverLegacyUnsafeWorkspaceExports(db, runIds).catch((err) => {
logger.warn({ err }, "Historical unsafe export recovery remains pending");
});
}
if (!options.skipObsoletePolicyReviews) {
timing.phase("obsolete_policy_reviews");
await dismissObsoleteNativePolicyReviews(db, runIds).catch((err) => {
logger.warn({ err }, "Obsolete native policy review lookup failed; continuing native reconciliation");
});
}
if (!options.skipAutomaticCompletionReviews) {
timing.phase("automatic_completion_reviews");
if (runIds?.length) {
const scopes = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations)
.where(inArray(nativeRunFinalizations.runId, runIds));
for (const scope of scopes) await dismissAutomaticCompletionReviews(db, scope.issueId);
} else {
await dismissAutomaticCompletionReviews(db);
}
}
timing.phase("candidate_scan");
const now = new Date();
const unscoped = !(runIds && runIds.length > 0);
const upkeepRunIds = unscoped ? await nextSettledCommittedUpkeep(db, now) : [];
const rows = await db
.select({
runId: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
issueId: nativeRunFinalizations.issueId,
issueStatus: issues.status,
issueStatusVersion: issues.statusVersion,
issueDecisionId: issues.lastStatusDecisionId,
coordinatorPhase: nativeRunFinalizations.phase,
runProjectionPending: options.skipNativeRunProjection ? pendingNativeRunProjectionCondition(db) : sql<boolean>`false`,
resultId: nativeRunFinalizations.resultId,
assessmentId: nativeRunFinalizations.assessmentId,
decisionId: nativeRunFinalizations.decisionId,
nativeWorkspaceSync: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
nativeStatusWakeIntentId: sql<unknown>`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`,
})
.from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.innerJoin(issues, and(
eq(issues.id, nativeRunFinalizations.issueId),
eq(issues.companyId, heartbeatRuns.companyId),
))
.where(and(
eq(heartbeatRuns.runtimeMode, "native"),
isNotNull(nativeRunFinalizations.resultId),
notInArray(nativeRunFinalizations.phase, ["terminal_failure"]),
options.skipNativePendingEffects ? not(pendingNativeStatusEffectsCondition(db)) : undefined,
or(
isNull(nativeRunFinalizations.nextAttemptAt),
lte(nativeRunFinalizations.nextAttemptAt, new Date()),
),
or(
isNull(nativeRunFinalizations.leaseOwner),
isNull(nativeRunFinalizations.leaseExpiresAt),
lte(nativeRunFinalizations.leaseExpiresAt, new Date()),
),
...(runIds && runIds.length > 0 ? [inArray(heartbeatRuns.id, runIds)] : []),
...(unscoped
? [upkeepRunIds.length > 0
? or(not(settledCommittedFinalization(now)), inArray(heartbeatRuns.id, upkeepRunIds))!
: not(settledCommittedFinalization(now))]
: []),
));
timing.candidates(rows.length);
timing.phase("candidates");
const results = [];
for (const row of rows) {
const pendingEffects = row.decisionId
? await db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
eq(statusDecisionEffects.companyId, row.companyId),
eq(statusDecisionEffects.issueId, row.issueId),
eq(statusDecisionEffects.decisionId, row.decisionId),
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
))
: [];
if (row.decisionId && pendingEffects.length > 0) {
if (options.skipNativePendingEffects) continue;
const decision = resolveNativeReconciliationStatus({
facts: { undeliveredEffectCount: pendingEffects.length },
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus,
agentId: row.agentId,
});
if (!decision.effects.some((effect) => effect.kind === "dispatch_pending_effect")) {
throw new Error("native_reconciliation_pending_effect_policy_missing");
}
const deliveredEffectIds = await dispatchPendingNativeStatusEffects({
db,
companyId: row.companyId,
issueId: row.issueId,
decisionId: row.decisionId,
runId: row.runId,
});
results.push({ runId: row.runId, phase: "committed", deliveredEffectIds });
continue;
}
if (row.assessmentId) {
const currentDecision = !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId
? await db.select().from(statusDecisions).where(and(
eq(statusDecisions.id, row.decisionId), eq(statusDecisions.companyId, row.companyId),
eq(statusDecisions.issueId, row.issueId),
)).limit(1).then(entries => entries[0] ?? null)
: null;
if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) {
// A later decision can supersede this run's task status, but cannot
// erase its accepted, still-authorized response. Repair presentation
// before the status-only early return below, without rerunning work.
await repairCommittedNativeChatResponse(db, {
companyId: row.companyId, issueId: row.issueId, runId: row.runId,
});
}
if (
!options.skipNativeReviewResponse &&
row.coordinatorPhase === "committed" &&
row.decisionId &&
row.resultId &&
row.assessmentId &&
record(currentDecision?.decisionJson).externalChatReviewPresentation
) {
// A later chat turn may retain the same pending review. Recover the
// earlier response independently before status reconciliation skips
// its superseded decision; this cannot change issue disposition.
await repairCommittedNativeReviewResponse(db, {
companyId: row.companyId,
issueId: row.issueId,
runId: row.runId,
decisionId: row.decisionId,
resultId: row.resultId,
assessmentId: row.assessmentId,
});
}
const source = { ...row, assessmentId: row.assessmentId };
if (options.skipNativeStatusReassessment) {
if (await nativeStatusReassessmentSourceIsSuperseded(db, source)) continue;
} else {
const reassessed = await reassessNativeStatusFromSource(db, source);
if (reassessed.kind !== "unchanged") {
if (reassessed.kind === "committed") results.push(reassessed.result);
continue;
}
}
}
if (options.skipNativeRunProjection && row.coordinatorPhase === "committed" && row.runProjectionPending) continue;
const barrier = await db.select({ status: workspaceOperations.status })
.from(workspaceOperations)
.where(and(
eq(workspaceOperations.companyId, row.companyId),
eq(workspaceOperations.issueId, row.issueId),
eq(workspaceOperations.heartbeatRunId, row.runId),
eq(workspaceOperations.phase, "workspace_finalize"),
))
.orderBy(desc(sql`${workspaceOperations.status} = 'succeeded'`), desc(workspaceOperations.createdAt))
.limit(1)
.then((entries) => entries[0] ?? null);
if (!barrier || barrier.status !== "succeeded") {
if (options.skipNativeWorkspaceFinalization) continue;
const recoveryDecision = resolveNativeReconciliationStatus({
facts: { workspaceOperationPending: true },
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus,
agentId: row.agentId,
});
if (!recoveryDecision.effects.some((effect) => effect.kind === "resume_workspace_operation")) {
throw new Error("native_reconciliation_workspace_resume_policy_missing");
}
const outcome = await attemptNativeWorkspaceFinalization({
db,
runId: row.runId,
environmentRuntime: options.environmentRuntime,
pluginWorkerManager: options.pluginWorkerManager,
});
// Busy is ownership, not another failed export or retry-budget debit.
if (outcome.kind !== "completed" && outcome.kind !== "failed") continue;
if (outcome.kind === "failed" || outcome.operation.status !== "succeeded") {
const failure = await recordNativeWorkspaceFinalizationFailure(db, row.runId, outcome.operation);
results.push({
...failure,
reconciliationAction: "resume_workspace_operation" as const,
...(outcome.kind === "completed" ? { workspaceOperationId: outcome.operation.id } : {}),
workspaceFinalizeStatus: "failed" as const,
reconciliationDecision: recoveryDecision,
});
if (failure.phase === "terminal_failure") {
await options
.onWorkspaceSettled?.({
runId: row.runId,
companyId: row.companyId,
agentId: row.agentId,
succeeded: false,
})
.catch(() => undefined);
}
continue;
}
const { operation } = outcome;
const workspaceFinalizeStatus = "succeeded" as const;
if (options.skipNativeResultCommit && row.coordinatorPhase !== "committed") continue;
try {
const finalized = await finalizeNativeRun({
db,
runId: row.runId,
workspaceFinalizeStatus,
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
skipNativeReviewResponse: options.skipNativeReviewResponse,
skipCommittedRunProjection: options.skipNativeRunProjection,
skipExternalConversationSettlement: options.skipExternalConversationSettlement,
skipExternalAgentSettlement: options.skipExternalAgentSettlement,
skipNativeRecoveryNotices: options.skipNativeRecoveryNotices,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
),
),
});
results.push({
...finalized,
reconciliationAction: "resume_workspace_operation" as const,
workspaceOperationId: operation.id,
workspaceFinalizeStatus,
reconciliationDecision: recoveryDecision,
});
if (
workspaceFinalizeStatus === "succeeded" &&
finalized.phase === "committed"
) {
await cleanupNativeWorkspaceSync(row.runId).catch(() => undefined);
await options
.onWorkspaceSettled?.({
runId: row.runId,
companyId: row.companyId,
agentId: row.agentId,
succeeded: true,
})
.catch(() => undefined);
}
} catch (error) {
const failure = await recordNativeFinalizationFailure({
db,
runId: row.runId,
error,
projectRunStatus: true,
});
results.push({
...failure,
reconciliationAction: "resume_workspace_operation" as const,
workspaceOperationId: operation.id,
workspaceFinalizeStatus,
reconciliationDecision: recoveryDecision,
});
}
continue;
}
if (options.skipNativeResultCommit && row.coordinatorPhase !== "committed") continue;
const replayDecision = resolveNativeReconciliationStatus({
facts: { canonicalReplay: true },
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus,
agentId: row.agentId,
});
if (replayDecision.reasonCode !== "completion_contract_satisfied") {
throw new Error("native_reconciliation_replay_policy_invalid");
}
try {
const finalized = await finalizeNativeRun({
db,
runId: row.runId,
workspaceFinalizeStatus: barrier.status as "succeeded" | "failed",
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
skipNativeReviewResponse: options.skipNativeReviewResponse,
skipCommittedRunProjection: options.skipNativeRunProjection,
skipExternalConversationSettlement: options.skipExternalConversationSettlement,
skipExternalAgentSettlement: options.skipExternalAgentSettlement,
skipNativeRecoveryNotices: options.skipNativeRecoveryNotices,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
),
),
});
results.push(finalized);
if (finalized.phase === "committed") {
await cleanupNativeWorkspaceSync(row.runId).catch(() => undefined);
await options
.onWorkspaceSettled?.({
runId: row.runId,
companyId: row.companyId,
agentId: row.agentId,
succeeded: true,
})
.catch(() => undefined);
}
} catch (error) {
results.push(await recordNativeFinalizationFailure({
db,
runId: row.runId,
error,
projectRunStatus: true,
}));
}
}
return results;
}
/** Existing settled-work topics whose producers write registered kinds:
* run lifecycle and controller renewal, environment leases, settled native
* processes, legacy shutdown fences, native process sessions, deferred
* receipts, stale issue locks, queued dispatch and accounting. */
const OWNERSHIP_REPORT_WAKE_QUEUES = [
DELIVERY_QUEUES.activeExecutionRecovery,
DELIVERY_QUEUES.orphanActiveLeases,
DELIVERY_QUEUES.settledNativeProcesses,
DELIVERY_QUEUES.legacyShutdownSettlement,
DELIVERY_QUEUES.nativeProcessSessions,
DELIVERY_QUEUES.deferredExecutionReceipts,
DELIVERY_QUEUES.staleIssueLocks,
DELIVERY_QUEUES.queuedRunDispatch,
DELIVERY_QUEUES.accounting,
// Company and agent pause changes: the hold-release watch of the deferred
// work-intent backstop (section E).
DELIVERY_QUEUES.agentLifecycle,
] as const;
/** Interval between report passes while in-progress records keep changing. */
export const OWNERSHIP_REPORT_INTERVAL_MS = 30_000;
/**
* The ownership reconciliation pass of this reconciler (plan section C). It
* classifies the owner of every registered in-progress record and records
* invariant violations. Kinds moved to the shared path in Phase 2 register a
* `settle`, which the pass calls only when the owner is proven gone; every
* other kind is report-only and never changed.
*
* Trigger and quiet condition (rules/sleep-aware-background-work.mdc):
* - one pass when the worker registers at startup, from durable truth;
* - a pass after any settled `active-execution-recovery` signal, which run
* lifecycle, controller renewal and process-evidence writes already emit,
* coalesced to at most one pass per 30 s;
* - a 30 s follow-up only while the last pass saw in-progress records whose
* set changed, or owners proven gone for the first time (rule I2 needs a
* second pass);
* - the end of rule I7's grace window for an issue that is still inside it.
* A pass that finds nothing in progress, or the same stuck set twice, sets no
* deadline: an idle instance runs no SQL here. The pass runs in the shared
* scheduler only, never on the path before `turn.start`.
*/
/** Section E backstop: a deferred intent whose hold cleared without a
* release event is promoted through the one promotion function. Promotion
* re-runs normal admission, so a wrong report never bypasses a gate. */
export async function applyOwnershipReportBackstops(db: Db, report: OwnershipReport): Promise<{ retryNeeded: boolean }> {
let retryNeeded = false;
// Rules: an undelivered user comment newer than the task's last run gets a
// wake, and a comment a stale continuation dropped reopens its done task.
// Save the missing outbox entry, then promote it.
for (const violation of report.violations.filter(row =>
row.reason === USER_COMMENT_WITHOUT_WAKE || row.reason === USER_COMMENT_DROPPED_BY_STALE_RUN)) {
if (!violation.companyId || !violation.issueId) continue;
const lost = violation.reason === USER_COMMENT_WITHOUT_WAKE;
try {
const [issue] = await db.select({ assigneeAgentId: issues.assigneeAgentId }).from(issues)
.where(and(eq(issues.companyId, violation.companyId), eq(issues.id, violation.issueId)));
if (!issue?.assigneeAgentId) continue;
const saved = await saveUndeliveredUserComments(db, { companyId: violation.companyId, issueId: violation.issueId,
agentId: issue.assigneeAgentId, commentIds: [violation.recordId], allowedStatuses: lost ? ["todo", "in_progress", "done"] : ["done"],
...(lost ? { executionWait: deferredWorkIntentWait("wake_lost", "This message's wake was lost; it is sent now."),
reopenAgentClosedDone: true } : {}) });
if (saved) await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: violation.companyId!,
trigger: "reconciler_backstop", issueId: violation.issueId }));
} catch (error) {
// The message still has no saved receipt: keep a pass due.
retryNeeded = true;
logger.warn({ err: error, commentId: violation.recordId }, "could not save a user comment without a wake");
}
}
const companies = new Set(report.violations.filter(violation => violation.reason === DEFERRED_INTENT_NOT_PROMOTED)
.flatMap(violation => violation.companyId ? [violation.companyId] : []));
for (const companyId of companies) {
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId, trigger: "reconciler_backstop" }))
.catch(error => { retryNeeded = true; logger.warn({ err: error, companyId }, "deferred intent backstop promotion failed"); });
}
const undispatched = new Set(report.newViolations.filter(violation => violation.reason === QUEUED_RUN_NOT_DISPATCHED)
.flatMap(violation => violation.companyId ? [violation.companyId] : []));
for (const companyId of undispatched) {
await db.transaction(tx => notifyQueuedRunDispatch(tx, { companyId }))
.catch(error => { retryNeeded = true; logger.warn({ err: error, companyId }, "queued run dispatch backstop failed"); });
}
return { retryNeeded };
}
export function registerOwnershipReconciliationWork(
db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
canRun: () => boolean,
options: {
onReport?: (report: OwnershipReport) => void;
/** False while heartbeat scheduling is suppressed for this instance. */
enabled?: () => Promise<boolean>;
} = {},
) {
const state = createOwnershipReportState();
let nextRunAt: number | null = null;
let lastFingerprint: string | null = null;
let lastReport: OwnershipReport | null = null;
// A wake is consumed only by a pass that actually ran, so a wake that lands
// during idle drain survives until dispatch reopens.
let pendingWake = true;
// While scheduling is suppressed the worker parks with no deadline and runs
// no SQL; the next producer signal re-checks suppression.
let parked = false;
let lastPassAt: number | null = null;
// True when the only reason a pass is due is the settle watch: the next run
// re-checks the settling kinds alone, not every kind and invariant.
let watchOnly = false;
// Rule I7's grace deadline from the last full pass; the light re-check keeps
// it and runs a full pass once it is due.
let graceDueAt: number | null = null;
const unsubscribers: Array<() => void> = [];
const worker = coordinator.register(DELIVERY_QUEUES.ownershipInvariantReport, {
retryMs: OWNERSHIP_REPORT_INTERVAL_MS,
// A wake is handled at once, so a suppressed instance parks immediately;
// full passes are throttled below without SQL.
coalesceMs: 0,
pausePassiveMaintenance: true,
run: async () => {
if (!canRun()) return;
if (options.enabled && !(await options.enabled())) {
parked = true;
nextRunAt = null;
return;
}
parked = false;
// At most one full pass per interval; a wake inside it waits for the
// interval's end (in memory, no SQL).
if (lastPassAt !== null && Date.now() - lastPassAt < OWNERSHIP_REPORT_INTERVAL_MS) {
nextRunAt = lastPassAt + OWNERSHIP_REPORT_INTERVAL_MS;
return;
}
if (watchOnly && !pendingWake && (graceDueAt === null || Date.now() < graceDueAt)) {
// Bounded and cheap: indexed listings of the settling kinds only.
lastPassAt = Date.now();
const recheck = await recheckSettlingKinds(db);
// A settle changes the set; the next full pass reports it.
if (recheck.settled.length > 0) { watchOnly = false; nextRunAt = Date.now(); return; }
if (recheck.watched === 0) watchOnly = false;
const watchDue = recheck.watched > 0 ? Date.now() + OWNERSHIP_REPORT_INTERVAL_MS : null;
nextRunAt = watchDue === null ? graceDueAt : graceDueAt === null ? watchDue : Math.min(watchDue, graceDueAt);
return;
}
lastPassAt = Date.now();
pendingWake = false;
const report = await runOwnershipReconciliationPass(db, { state });
const backstop = await applyOwnershipReportBackstops(db, report);
const fingerprint = ownershipReportFingerprint(report);
const changing = report.inProgress > 0 && fingerprint !== lastFingerprint;
// A settling kind's live owner can die without writing a signal, so
// keep one bounded re-check due while such records exist. A backstop
// that could not save or promote keeps one due as well.
const fullPassDue = changing || report.awaitingSecondProof > 0 || report.retryNeeded || backstop.retryNeeded;
watchOnly = !fullPassDue && report.settleWatch > 0;
const followUp = fullPassDue || report.settleWatch > 0
? Date.now() + OWNERSHIP_REPORT_INTERVAL_MS : null;
// A record inside a grace window (rule I7) is a real future obligation.
const graceDue = report.nextDueAt?.getTime() ?? null;
graceDueAt = graceDue;
nextRunAt = followUp === null ? graceDue : graceDue === null ? followUp : Math.min(followUp, graceDue);
if (fingerprint !== lastFingerprint) {
// One line per change of the in-progress set: the verdict counts per
// kind are the Phase 1 data (server log only, never Telemetry).
logger.info({
inProgress: report.inProgress,
violations: report.violations.length,
kinds: Object.fromEntries(report.kinds.filter(kind => kind.inProgress > 0 || kind.error)
.map(kind => [kind.kind, { ...kind.verdicts, ...(kind.settled ? { settled: kind.settled } : {}),
...(kind.truncated ? { truncated: true } : {}),
...(kind.error ? { error: kind.error } : {}) }])),
}, "ownership reconciliation pass");
}
lastFingerprint = fingerprint;
lastReport = report;
options.onReport?.(report);
},
hasPending: async () => pendingWake && !parked,
nextRunAt: () => nextRunAt,
onStop: () => { for (const unsubscribe of unsubscribers.splice(0)) unsubscribe(); },
});
// Producers of the registered kinds that already publish settled work
// signals. Kinds whose writers publish none are read on any of these
// wakes and at startup (named in the report's coverage gaps).
for (const queue of OWNERSHIP_REPORT_WAKE_QUEUES) {
unsubscribers.push(subscribeDeliveryWork(db, queue, () => { pendingWake = true; parked = false; worker.wake(); }));
}
return {
ready: worker.ready,
wake: worker.wake,
lastReport: () => lastReport,
stop: worker.stop,
};
}