mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues check assigned tasks for work that has no
live run and no queued wake. Since #21872 they load only changed rows,
and they save pending deferrals so a replaced server keeps their
deadlines
> - Two costs remained. A running task's check defers to now + 30 s, and
each new deadline rewrote its saved deferral row: about two writes a
minute per running task per queue
> - Also, two writers that settle an agent after its runs end set the
agent's status and `updated_at` but not `last_heartbeat_at`. The
changed-rows agent rule then reads the write as a pause or resume and
re-checks all of that agent's tasks for about 5 minutes
> - This pull request keeps a repeatedly deferring task's saved row
until its deadline is well past, and makes both writers record a
heartbeat as `finalizeAgentStatus` does
> - The benefit is near-zero deferral writes for running tasks and no
whole-backlog re-check after an abandoned or stranded run settles an
agent
## Linked Issues or Issue Description
Refs #21872, #21861
**What happened?**
1. In `recovery/work.ts`, a deferring check upserted its
`issue_recovery_deferrals` row whenever the deadline changed. A deferral
to now + 30 s changes on every check, so a task deferring for 20 minutes
wrote its row 41 times. At 1,000 concurrent agents that is roughly 30 to
70 writes a second.
2. `execution-control-reconciliation.ts` (abandoned run past its
execution-control deadline) and `heartbeat/agent-status.ts`
`reconcileStrandedAgentStatuses` (agent left "running" with no live run)
set the agent to idle with `updated_at` but without `last_heartbeat_at`.
The agent clause of `changedTaskCondition` matches an agent that is not
running and whose `updated_at` is newer than its last heartbeat. So each
such write made the backstop re-check every task of that agent for about
5 minutes.
**Expected behavior**
A task that keeps deferring writes its row a small fixed number of
times. Settling an agent after its runs end does not re-check the
agent's other tasks; the run's own task is still checked promptly.
**Steps to reproduce**
1. Defer one task's check to now + 30 s on every check for 20 simulated
minutes and count inserts into `issue_recovery_deferrals`: 41.
2. Seed an agent left "running" with 140 settled reviews, settle it with
`reconcileAbandonedExecutionControl` or
`reconcileStrandedAgentStatuses`, and count stranded checks of those
reviews over the next backstop: 280 per agent.
**Paperclip version or commit**
`master` at b4fa1c7fb1.
## What Changed
- `recovery/work.ts`: a deferring check skips the upsert while this
server knows a saved row for the task whose deadline is less than
`DEFERRAL_REFRESH_MS` (5 minutes) past. It refreshes the row after that.
The refresh comes before the 10-minute stale prune margin, so a
still-deferring task's row never looks stale. A boot re-checks every
task with a row whatever its deadline, so the exact deadline only
matters to the conditional deletes and the stale prune, which still
match the value this server read or wrote. A cleared row, a failed write
or a reload still drop the server's memory of the row, so the next
deferral writes it again.
- `execution-control-reconciliation.ts` and `heartbeat/agent-status.ts`:
settling an agent after its runs end sets `lastHeartbeatAt` with
`updatedAt`, as `finalizeAgentStatus` already does, through a new
`settledHeartbeat(now)`. It records the settlement time only when the
agent already has a heartbeat; an agent with none keeps none, so the
first real heartbeat stays first (the existing "repairs a shutdown flag
without inventing a heartbeat" test still passes).
- `recovery/changed-tasks.ts`: the comment now states the contract:
every write that settles an agent's status after its runs end sets
`last_heartbeat_at`.
- Tests: a write-rate census in
`issue-recovery-saved-deferrals.test.ts`, and an agent-settle census and
guard in `issue-recovery-company-reload-census.test.ts`.
Why fix the writers, not the rule: the agent rule's discriminator
already is `last_heartbeat_at` against `updated_at` ("a run's finish
records a heartbeat"). Pauses, resumes, holds and configuration writes
leave `last_heartbeat_at` alone; run settlement records one. No other
column tells a settle-to-idle apart from a resume to idle, because both
end with status idle and a new `updated_at`. Since #21872, lifecycle
transitions also send their own agent hint, but configuration writes to
an idle agent still rely on this rule (the #21861 guard). So the sound
fix is for the two run-settling writers to follow the same contract as
`finalizeAgentStatus`. Recording the heartbeat at settlement has the
same effect on agent timers as a normal run finish.
## Verification
Red/green:
| Census | Before (master code) | After |
| --- | --- | --- |
| One task deferring every 30 s for 20 minutes: deferral writes | 41 (41
checks) | 4 (41 checks) |
| Same task: saved deadline behind now after 20 minutes | 0 (rewritten
each check) | 3 min (refreshed; the test fails above 6 min, and fails
with the refresh removed: 1 write, 19.5 min) |
| Two agents (140 settled reviews each) settled by an abandoned-run
reconcile and a stranded-status reconcile: backlog checks over the next
backstops | 560 | 0 |
| Same census: statements | 12,570 | 2,507 |
Guards:
- The abandoned run's task is checked within 1 minute of the reconcile
(passes before and after).
- The three deferral tests from #21872 still pass: replacement mid-wait
checks at the original deadline, the two-server interleaving keeps
server B's deadline, and the stale row of a task that left the queue is
removed.
Focused suites, each file run alone in the foreground:
`issue-recovery-company-reload-census.test.ts` 8 passed (38 s),
`issue-recovery-saved-deferrals.test.ts` 6 passed (14 s),
`heartbeat/agent-status.test.ts` and
`execution-control-reconciliation.test.ts` 60 passed,
`issue-recovery-idle-backstop.test.ts` 4 passed. An 18-file run earlier:
516 passed; the 4 failures were 1 test that this PR then fixed
(heartbeat for agents with none) and 3 in
`issue-recovery-idle-backstop.test.ts`, which pass alone (see Risks).
Server `tsc --noEmit`: passed. `pnpm verify`: not run; required CI
covers the broad suites.
## Risks
- A task that keeps deferring may have a saved deadline up to 5 minutes
old. Only the stale prune reads it, and it deletes a row only after the
task left the queue and the deadline is 10 minutes past, so a
still-deferring task's row is refreshed first.
- If another server deletes the row (its check stopped deferring and it
matched the same deadline) while this server keeps deferring, this
server rewrites the row at its next refresh (at most about 5 minutes)
instead of at its next check. Within that window a crash of this server
would leave the task to its successor's next changed-rows or full pass.
This needs the two servers to disagree about the same task, which only
happens through facts held in one server's memory.
- `last_heartbeat_at` now also moves when an abandoned or stranded run
settles the agent. Agent timers then count from that time, as they do
after a normal run finish.
- An agent that never recorded a heartbeat still has its whole backlog
re-checked once when such a writer settles it. This is rare: that agent
has not finished a run yet.
- Not from this PR: `issue-recovery-idle-backstop.test.ts` > "picks up a
finished run, a user message, a raised budget and a resumed agent within
a minute" hung once and failed 3 tests when run with other recovery
files in one process. It passes alone. Its waits have no time limits,
and its fixture agents can take on-demand wakes, which is the hang
pattern that #21867 removed from the agent-row census. It needs the same
treatment in a separate change.
- No schema change. Nothing on the run-start path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
380 lines
18 KiB
TypeScript
380 lines
18 KiB
TypeScript
import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js";
|
|
import { notifyExternalAgentSettlement } from "../modules/external-agents/index.js";
|
|
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
|
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
|
|
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm";
|
|
import { settledHeartbeat } from "./heartbeat/agent-status.js";
|
|
import {
|
|
agents,
|
|
heartbeatRuns,
|
|
issues,
|
|
nativeRunFinalizations,
|
|
type Db,
|
|
} from "@paperclipai/db";
|
|
import { parseIssueExecutionState } from "./issue-execution-policy.js";
|
|
import { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
|
import { hasLiveLegacyController } from "./legacy-controller-lease.js";
|
|
import { reportRunFailure } from "./run-failure-report.js";
|
|
import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js";
|
|
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
|
|
import { EXECUTION_FINALIZATION_DEADLINE_EXCEEDED, EXECUTION_FINALIZATION_DEADLINE_MESSAGE,
|
|
EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { NATIVE_STARTUP_CLEANUP_PENDING, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-runtime/native-startup-cleanup.js";
|
|
import { releaseAbandonedNativeOwners } from "./native-runtime/native-failed-run-owner-release.js";
|
|
|
|
/** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */
|
|
export async function reconcileAbandonedExecutionControl(
|
|
db: Db,
|
|
now = new Date(),
|
|
options: { startupCleanup?: (scope: { companyId: string; runId: string }) => Promise<unknown> } = {},
|
|
) {
|
|
const nativeDue = await db
|
|
.select({
|
|
runId: nativeRunFinalizations.runId,
|
|
issueId: nativeRunFinalizations.issueId,
|
|
companyId: nativeRunFinalizations.companyId,
|
|
failureCode: nativeRunFinalizations.failureCode,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner,
|
|
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
|
attempt: nativeRunFinalizations.attempt,
|
|
})
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
isNotNull(nativeRunFinalizations.controlDeadlineAt),
|
|
lte(nativeRunFinalizations.controlDeadlineAt, now),
|
|
),
|
|
)
|
|
.orderBy(asc(nativeRunFinalizations.controlDeadlineAt), asc(nativeRunFinalizations.runId))
|
|
.limit(50);
|
|
const controlDue = await db
|
|
.select({
|
|
runId: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
context: heartbeatRuns.contextSnapshot,
|
|
nativeIssueId: nativeRunFinalizations.issueId,
|
|
failureCode: nativeRunFinalizations.failureCode,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner,
|
|
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
|
attempt: nativeRunFinalizations.attempt,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.leftJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
|
.where(
|
|
and(
|
|
isNotNull(heartbeatRuns.executionControlDeadlineAt),
|
|
lte(heartbeatRuns.executionControlDeadlineAt, now),
|
|
),
|
|
)
|
|
.orderBy(asc(heartbeatRuns.executionControlDeadlineAt), asc(heartbeatRuns.id))
|
|
.limit(50);
|
|
const due = [
|
|
...new Map(
|
|
[
|
|
...nativeDue,
|
|
...controlDue.map((row) => ({
|
|
...row,
|
|
issueId: row.nativeIssueId ?? (
|
|
typeof row.context?.issueId === "string"
|
|
? row.context.issueId
|
|
: null),
|
|
})),
|
|
].map((row) => [row.runId, row]),
|
|
).values(),
|
|
];
|
|
let surfaced = 0;
|
|
// Bound contention latency across independent tasks: a locked task must not
|
|
// consume the entire reconciliation window for every task behind it.
|
|
let nextCandidate = 0;
|
|
await Promise.all(Array.from({ length: Math.min(5, due.length) }, async () => {
|
|
while (nextCandidate < due.length) {
|
|
const candidate = due[nextCandidate++]!;
|
|
try {
|
|
if (candidate.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) {
|
|
if (!options.startupCleanup) continue;
|
|
try {
|
|
await options.startupCleanup({ companyId: candidate.companyId, runId: candidate.runId });
|
|
} finally {
|
|
await db.transaction(async tx => {
|
|
if (candidate.issueId) await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, candidate.issueId)).for("update");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, candidate.runId)).for("update");
|
|
if (owner?.failureCode !== NATIVE_STARTUP_CLEANUP_PENDING || !owner.controlDeadlineAt || owner.controlDeadlineAt > now ||
|
|
owner.leaseOwner !== candidate.leaseOwner || owner.controllerGeneration !== candidate.controllerGeneration || owner.attempt !== candidate.attempt) return;
|
|
const cleanupRetryAt = new Date(Math.max(now.getTime(), Date.now()) + EXECUTION_RECONCILIATION_INTERVAL_MS);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
|
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: cleanupRetryAt }).where(eq(nativeRunFinalizations.runId, candidate.runId));
|
|
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: cleanupRetryAt }).where(and(eq(heartbeatRuns.id, candidate.runId),
|
|
isNotNull(heartbeatRuns.executionControlDeadlineAt), lte(heartbeatRuns.executionControlDeadlineAt, now)));
|
|
});
|
|
}
|
|
continue;
|
|
}
|
|
// Set inside the transaction only when the write below genuinely
|
|
// transitions the run into "failed". Read after the transaction
|
|
// commits, so a rolled-back write never reports a false failure.
|
|
let terminalRunToReport: typeof heartbeatRuns.$inferSelect | null = null;
|
|
const repaired = await db.transaction(async (tx) => {
|
|
await tx.execute(
|
|
sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`,
|
|
);
|
|
const task = candidate.issueId
|
|
? (
|
|
await tx
|
|
.select()
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, candidate.issueId),
|
|
eq(issues.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.for("update")
|
|
)[0]
|
|
: null;
|
|
const [coordinator] = await tx
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.runId, candidate.runId),
|
|
eq(nativeRunFinalizations.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const [run] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, candidate.runId),
|
|
eq(heartbeatRuns.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.for("update");
|
|
if (
|
|
!run ||
|
|
![
|
|
coordinator?.controlDeadlineAt,
|
|
run.executionControlDeadlineAt,
|
|
].some((deadline) => deadline && deadline <= now)
|
|
)
|
|
return false;
|
|
if (coordinator?.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) return false;
|
|
const startupContext = coordinator && readNativeStartupContext(run, coordinator);
|
|
if (startupContext) {
|
|
if (!await promoteNativeStartupCleanup(tx as unknown as Db, run, coordinator!, now)) {
|
|
const retryAt = new Date(Math.max(now.getTime() + EXECUTION_RECONCILIATION_INTERVAL_MS,
|
|
startupContext.deadlineAtMs, coordinator!.leaseExpiresAt?.getTime() ?? 0));
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
|
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: retryAt }).where(eq(nativeRunFinalizations.runId, run.id));
|
|
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: retryAt }).where(eq(heartbeatRuns.id, run.id));
|
|
}
|
|
return false;
|
|
}
|
|
if (coordinator && !coordinator.resultId && run.status === "running") {
|
|
const nativeOwnerLeaseAt = coordinator.leaseOwner && coordinator.leaseExpiresAt && coordinator.leaseExpiresAt > now
|
|
? coordinator.leaseExpiresAt : null;
|
|
const sameRunResumeScheduled = coordinator.phase === "retryable_failure" && !coordinator.leaseOwner;
|
|
const nativeSettlementDue = Boolean(coordinator.controlDeadlineAt && coordinator.controlDeadlineAt <= now);
|
|
if (!nativeSettlementDue && (nativeOwnerLeaseAt || sameRunResumeScheduled)) {
|
|
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: null }).where(eq(heartbeatRuns.id, run.id));
|
|
return false;
|
|
}
|
|
if (nativeOwnerLeaseAt) {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
|
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: nativeOwnerLeaseAt })
|
|
.where(eq(nativeRunFinalizations.runId, run.id));
|
|
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: run.executionControlDeadlineAt ? nativeOwnerLeaseAt : null })
|
|
.where(eq(heartbeatRuns.id, run.id));
|
|
return false;
|
|
}
|
|
}
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ executionControlDeadlineAt: null })
|
|
.where(eq(heartbeatRuns.id, run.id));
|
|
if (
|
|
[
|
|
"succeeded",
|
|
"failed",
|
|
"cancelled",
|
|
"timed_out",
|
|
"interrupted",
|
|
].includes(run.status)
|
|
) {
|
|
if (coordinator?.controlDeadlineAt)
|
|
await tx
|
|
.update(nativeRunFinalizations)
|
|
.set({ controlDeadlineAt: null })
|
|
.where(eq(nativeRunFinalizations.runId, run.id));
|
|
return true;
|
|
}
|
|
// A persisted result belongs to the existing finalizer, not a replacement provider.
|
|
if (coordinator?.resultId) {
|
|
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
|
|
await tx
|
|
.update(nativeRunFinalizations)
|
|
.set({
|
|
controlDeadlineAt: null,
|
|
leaseOwner: null,
|
|
leaseExpiresAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(nativeRunFinalizations.runId, run.id));
|
|
return true;
|
|
}
|
|
if (await hasLiveLegacyController(tx as unknown as Db, run)) {
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ executionControlDeadlineAt: run.controllerLeaseExpiresAt })
|
|
.where(eq(heartbeatRuns.id, run.id));
|
|
return false;
|
|
}
|
|
const cause = EXECUTION_FINALIZATION_DEADLINE_EXCEEDED;
|
|
const nextAction = EXECUTION_FINALIZATION_DEADLINE_MESSAGE;
|
|
// The deadline replaces the run's error code. Keep the earlier
|
|
// diagnosis as its root cause, as the native finalizer does.
|
|
const priorFailureCode = coordinator?.failureCode ?? run.errorCode;
|
|
const recordsOriginalFailure = Boolean(priorFailureCode) && priorFailureCode !== cause &&
|
|
typeof run.resultJson?.originalFailureCode !== "string";
|
|
if (coordinator)
|
|
await tx
|
|
.update(nativeRunFinalizations)
|
|
.set({
|
|
phase: "terminal_failure",
|
|
controlDeadlineAt: null,
|
|
leaseOwner: null,
|
|
leaseExpiresAt: null,
|
|
recoveryState: "blocked",
|
|
nextAttemptAt: null,
|
|
failureCode: cause,
|
|
failureDetail: {
|
|
...coordinator.failureDetail,
|
|
nextAction,
|
|
recoveryOwner: { kind: "board" },
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(nativeRunFinalizations.runId, run.id));
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyNativeEnvironmentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, runId: run.id });
|
|
const [updatedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "failed",
|
|
executionStatusDeliveryId: randomUUID(),
|
|
finishedAt: now,
|
|
...(coordinator
|
|
? { nativePhase: "terminal_failure", nativePhaseUpdatedAt: now }
|
|
: {}),
|
|
errorCode: cause,
|
|
error: nextAction,
|
|
nextAction,
|
|
...(recordsOriginalFailure
|
|
? { resultJson: { ...run.resultJson, originalFailureCode: priorFailureCode } }
|
|
: {}),
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(heartbeatRuns.id, run.id))
|
|
.returning();
|
|
if (updatedRun && updatedRun.status !== run.status) {
|
|
terminalRunToReport = updatedRun;
|
|
}
|
|
await tx
|
|
.update(agents)
|
|
// A run settling records a heartbeat, as finalizeAgentStatus does, so
|
|
// the changed-rows agent rule (recovery/changed-tasks.ts) does not
|
|
// read it as a pause or resume and re-check the agent's whole backlog.
|
|
.set({ status: "idle", lastHeartbeatAt: settledHeartbeat(now), updatedAt: now })
|
|
.where(
|
|
and(
|
|
eq(agents.id, run.agentId),
|
|
eq(agents.companyId, run.companyId),
|
|
eq(agents.status, "running"),
|
|
eq(agents.lifecycleState, "ready"),
|
|
sql`not exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.agentId} = ${run.agentId} and ${heartbeatRuns.status} = 'running')`,
|
|
),
|
|
);
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(run.contextSnapshot);
|
|
const nativeReviewAssignment = nativeReviewContext && task
|
|
? await getNativeReviewAssignment(tx as unknown as Db, {
|
|
companyId: run.companyId,
|
|
issueId: task.id,
|
|
agentId: run.agentId,
|
|
contextSnapshot: nativeReviewContext,
|
|
})
|
|
: null;
|
|
const review = task?.status === "in_review" ? parseIssueExecutionState(task.executionState) : null;
|
|
const isCurrentReviewer = (review?.status === "pending" &&
|
|
review.currentParticipant?.type === "agent" && review.currentParticipant.agentId === run.agentId)
|
|
|| nativeReviewAssignment?.interaction.status === "pending";
|
|
if (
|
|
!task ||
|
|
(task.assigneeAgentId !== run.agentId && !isCurrentReviewer) ||
|
|
["done", "cancelled"].includes(task.status) ||
|
|
(task.executionRunId && task.executionRunId !== run.id) ||
|
|
(task.checkoutRunId && task.checkoutRunId !== run.id)
|
|
) {
|
|
if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
|
|
return true;
|
|
}
|
|
await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id });
|
|
await tx
|
|
.update(issues)
|
|
.set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
|
|
.where(eq(issues.id, task.id));
|
|
await issueRecoveryActionService(
|
|
tx as unknown as Db,
|
|
).upsertSourceScoped({
|
|
companyId: run.companyId,
|
|
sourceIssueId: task.id,
|
|
kind: "active_run_watchdog",
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
returnOwnerAgentId: task.assigneeAgentId,
|
|
cause,
|
|
fingerprint: `execution-control:${run.id}`,
|
|
evidence: {
|
|
runId: run.id,
|
|
...(isCurrentReviewer ? { reviewParticipantAgentId: run.agentId } : {}),
|
|
originalFailureCode: coordinator?.failureCode ?? run.errorCode,
|
|
providerOwnership: "unverified",
|
|
},
|
|
nextAction,
|
|
wakePolicy: null,
|
|
maxAttempts: 3,
|
|
supersedeOnIdentityChange: true,
|
|
});
|
|
if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
|
|
return true;
|
|
});
|
|
if (repaired) surfaced += 1;
|
|
if (terminalRunToReport) void reportRunFailure(db, terminalRunToReport);
|
|
} catch {
|
|
logger.warn(
|
|
{ runId: candidate.runId },
|
|
"Execution finalization reconciliation remains pending; continuing with other runs",
|
|
);
|
|
}
|
|
}
|
|
}));
|
|
// Owners left on runs that already ended outside the native executor have no
|
|
// control deadline; their expired lease is the due time.
|
|
const releasedOwners = await releaseAbandonedNativeOwners(db, now).catch((error: unknown) => {
|
|
logger.warn({ err: error }, "Abandoned native owner release remains pending");
|
|
return 0;
|
|
});
|
|
return { scanned: due.length, surfaced, releasedOwners };
|
|
}
|