fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 08:08:38 -06:00
1 parent 11840f9904
commit 6917d0568d
4 files changed
+108 -12

No files matched your search

@@ -1202,6 +1202,52 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id);
});
it("starts each claimed run before claiming the next, within the agent's concurrency cap", async () => {
// A claim commits a 60 s legacy controller lease that only the run's own
// execution renews. A batch whose later claims are slow must not hold the
// earlier claimed runs back from executing until their leases lapse.
const [company] = await db.insert(companies).values({ name: "Claim batch", issuePrefix: "CB" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Busy agent", role: "engineer", status: "idle",
adapterType: "process", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 2 } } }).returning();
const queued = [];
for (let index = 0; index < 3; index++) {
const [issue] = await db.insert(issues).values({ companyId: company.id, title: `Batch task ${index}`, status: "todo",
assigneeAgentId: agent.id }).returning();
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment",
status: "claimed", claimedAt: new Date() }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "queued",
invocationSource: "assignment", wakeupRequestId: wake.id, responsibleUserId: "queue-test-user", contextSnapshot: { issueId: issue.id },
createdAt: new Date(Date.now() - (3 - index) * 1_000) }).returning();
await db.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() })
.where(eq(issues.id, issue.id));
queued.push(run);
}
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
let releaseSecondClaim!: () => void;
const secondClaimMayContinue = new Promise<void>(resolve => { releaseSecondClaim = resolve; });
let budgetChecks = 0;
deps.budgets.getInvocationBlock.mockImplementation(async () => {
budgetChecks += 1;
if (budgetChecks === 2) await secondClaimMayContinue;
return null;
});
const admission = vi.fn(() => false);
deps.isRunAdmissionClosed = admission;
const pass = createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id);
try {
await vi.waitFor(() => expect(budgetChecks).toBe(2));
// The first run is executing while the second claim is still in progress.
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(queued[0]!.id);
} finally { releaseSecondClaim(); }
const claimed = await pass;
expect(claimed.map(run => run.id)).toEqual([queued[0]!.id, queued[1]!.id]);
expect(deps.executeRun.mock.calls.map(([runId]) => runId)).toEqual([queued[0]!.id, queued[1]!.id]);
// The run-start admission gate is checked before every claim, and the cap holds.
expect(admission.mock.calls.length).toBeGreaterThanOrEqual(3);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued[2]!.id)))[0].status).toBe("queued");
});
it("leaves a queued run for the next server when shutdown closes admission during the queue scan", async () => {
const { run, agent, wake } = await fixture("queued");
await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
+11 -8
View File
@@ -2589,17 +2589,21 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
noteDeclinedAgedClaim(queuedRun, err.message);
continue;
}
if (claimed) {
claimedRuns.push(claimed);
dispatcher.claimed(queuedRun);
declinedAgedClaimAt.delete(queuedRun.id);
} else {
if (!claimed) {
noteDeclinedAgedClaim(queuedRun, "claim_declined");
continue;
}
claimedRuns.push(claimed);
dispatcher.claimed(queuedRun);
declinedAgedClaimAt.delete(queuedRun.id);
// Start each run as soon as it is claimed. The claim committed a legacy
// controller lease that only the run's own execution renews; waiting
// for the rest of a slow batch to be claimed let that lease lapse.
startClaimedRun(claimed);
}
if (claimedRuns.length === 0) return [];
return claimedRuns;
for (const claimedRun of claimedRuns) {
function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) {
const execution = executeRun(claimedRun.id).catch((err) => {
logger.error(
{ err, runId: claimedRun.id },
@@ -2617,7 +2621,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
activeRunExecutionPromises.delete(execution);
});
}
return claimedRuns;
}).finally(async () => {
// Dispatch promoted inputs after the agent start lock is released.
try {
@@ -189,6 +189,42 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
expect(await renewLegacyControllerLease(db, run)).toBe(false);
expect(await revokeExpiredLegacyController(db, run)).toBe(false);
});
it("renews at once when the watch starts on a lease older than one renewal interval", async () => {
// The watch can start long after the claim (a slow claim batch). Waiting for
// the first 10 s tick let a still-live lease reach its deadline unrenewed.
const run = await seed();
const [late] = await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp() + interval '1500 milliseconds'` })
.where(eq(heartbeatRuns.id, run.id)).returning();
const controller = new AbortController();
const watch = watchLegacyControllerLease(db, late, controller);
try {
await new Promise(resolve => setTimeout(resolve, 2_000));
expect(controller.signal.reason?.leaseLossTiming).toBeUndefined();
expect(controller.signal.aborted).toBe(false);
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(saved.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now() + LEGACY_CONTROLLER_LEASE_MS - 5_000);
expect(saved.controllerBootId).toBe(legacyControllerBootId);
} finally { watch.stop(); }
});
it.each(["revoked by recovery", "reaped", "lapsed"] as const)(
"a watch that starts late on a lease that was %s still loses it and never revives it", async outcome => {
const run = await seed();
await expire(run.id);
if (outcome === "revoked by recovery") expect(await revokeExpiredLegacyController(db, run)).toBe(true);
if (outcome === "reaped") await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const controller = new AbortController();
// The stale claim-time row still names this boot with a lease that has not expired locally.
const watch = watchLegacyControllerLease(db, { ...run, controllerLeaseExpiresAt: new Date(Date.now() + 1_000) }, controller);
try {
await vi.waitFor(() => expect(controller.signal.aborted).toBe(true), { timeout: 2_000 });
expect(controller.signal.reason).toBeInstanceOf(LegacyControllerLeaseLostError);
expect(controller.signal.reason.leaseLossCause).toBe("renewal_rejected");
const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(after).toMatchObject({ status: before.status, controllerBootId: before.controllerBootId,
controllerLeaseExpiresAt: before.controllerLeaseExpiresAt });
} finally { watch.stop(); }
});
it("rejects dispatch at the lease deadline even if the database query never settles", async () => {
const run = await seed();
const hungDb = fakeRenewalDb(() => new Promise(() => {}));
@@ -619,7 +655,9 @@ describe("legacy controller lease loss attribution", () => {
await new Promise((resolve) => setTimeout(resolve, 5));
expect(controller.signal.reason.leaseLossCause).toBe("renewal_deadline");
expect(controller.signal.reason.leaseLossTiming.deadlineLateMs).toBeGreaterThanOrEqual(60);
expect(controller.signal.reason.leaseLossTiming.renewalPendingMs).toBeNull();
// A watch that starts this close to the deadline renews at once; that renewal never answered.
expect(controller.signal.reason.leaseLossTiming.renewalPendingMs).toBeGreaterThanOrEqual(60);
expect(controller.signal.reason.leaseLossTiming.renewalCount).toBe(0);
} finally { watch.stop(); }
});
});
+12 -3
View File
@@ -390,7 +390,8 @@ export function watchLegacyControllerLease(
deadline = setTimeout(deadlinePassed, delayMs);
deadline.unref();
};
armDeadline(Math.max(0, (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now()));
const initialRemainingMs = (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now();
armDeadline(Math.max(0, initialRemainingMs));
const assertOwned = async (stage?: "dispatching") => {
if (stopped) return;
controller.signal.throwIfAborted();
@@ -447,11 +448,19 @@ export function watchLegacyControllerLease(
armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - renewedFrom)));
}
};
const timer = setInterval(() => {
const tick = () => {
if (pending || stopped) return;
pending = true;
void assertOwned().catch(() => lost("renewal_error")).finally(() => { pending = false; });
}, LEGACY_CONTROLLER_RENEW_MS);
};
const timer = setInterval(tick, LEGACY_CONTROLLER_RENEW_MS);
timer.unref();
// The watch can start well after the claim that wrote the lease (a slow
// batch of claims, a loaded database). A lease older than one renewal
// interval is renewed now rather than on the next tick. The renewal is the
// same conditional statement as every other one: it only extends a lease
// this boot still holds on a running row, so a lease that lapsed and was
// revoked, reaped or taken over is still lost.
if (initialRemainingMs < LEGACY_CONTROLLER_LEASE_MS - LEGACY_CONTROLLER_RENEW_MS) tick();
return { assertOwned, stop() { stopped = true; clearInterval(timer); clearTimeout(deadline); } };
}