mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server starts queued agent runs. A legacy-runtime run gets a 60 s controller lease when it is claimed. Only the run's own execution renews that lease. > - The queued-run dispatch claimed all the runs of one agent in a loop. It started their executions only after the last claim finished. > - When the database is slow (for example, while a flood of recovery runs fills the queue), one claim can take tens of seconds. The earlier claims' leases then expire before their executions start. The run fails with `controller_lease_lost` and `renewalCount: 0`. > - Also, the lease watch waited for its first 10 s tick before it renewed, even when the claim-time lease was almost spent. > - This pull request starts each run as soon as it is claimed. It also makes the lease watch renew immediately when the lease is older than one renewal interval. > - The benefit is that a slow claim batch no longer makes runs fail before they do any work. ## Linked Issues or Issue Description Refs #21127, #21042, #21222 **What happened?** On staging, legacy-runtime runs failed with `controller_lease_lost`. Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`, and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease watch started on a lease that had already expired, and no renewal was ever sent. In one run, the claim was at 23:20:02 and the loss was at 23:23:11. Losses came in groups on the same agent at the same moment. For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all lost at 23:23:11.6. **Why did it happen?** `startNextQueuedRunForAgent` claimed every run of a batch before it called `executeRun` for any of them. Each claim commits a 60 s lease, but nothing renews it until the run's execution starts its lease watch. A queue flood (fixed separately in #21222) made each claim slow, so a batch took minutes. This bug predates today's changes. The queue flood exposed it. The run-start hand-over (#21127) did not cause it: no renewal was queued on the controller-lease connection. **Expected behavior** A claimed run starts its execution, and renews its lease, without waiting for other claims. A lease watch that starts late renews at once. A lease that lapsed and was revoked, reaped or taken over is still lost and is never revived. ## What Changed - `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent` starts each run's execution right after its own claim, not after the whole batch. The concurrency cap, the agent start lock and the run-admission check before each claim are unchanged. If a later claim throws, the runs claimed before it are already executing, not stranded. After merging master, the loop walks master's queued-run dispatcher (#21234: person-initiated runs first, paced recovery delivery): each successful claim is reported to the dispatcher (`dispatcher.claimed`) and then started at once, so master's order and pacing are unchanged. Master's long-queued-run bookkeeping (#21349) is kept too: a claim that returns nothing is still recorded as declined (`noteDeclinedAgedClaim`), and a successful claim still clears that record before the run starts. - `server/src/services/legacy-controller-lease.ts`: `watchLegacyControllerLease` renews immediately when the claim-time lease has less than 50 s left (older than one renewal interval). It uses the same conditional renewal statement. That statement extends only a lease that this boot still holds on a running row. The deadline is unchanged. - Tests: red/green tests for both changes. A test also proves that a late watch on a revoked, reaped or lapsed lease still loses it and does not change the row. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass. - Red on master (source reverted, new tests kept): "starts each claimed run before claiming the next" fails (`executeRun` not called while the second claim is in progress). "renews at once when the watch starts on a lease older than one renewal interval" fails (aborted at the claim-time deadline). The three "starts late ... still loses it" cases fail on master because master loses them by deadline and does not check ownership. - After deploy: `controller_lease_lost` with `renewalCount: 0` should drop to about zero on staging, also with a full queue. A late watch now shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a silent expired start. ## Risks - Low. The executions of a batch now start while later claims of the same batch are still in progress. Execution does not claim runs and is not awaited by the loop, so the cap and the lock are not affected. - A late lease watch sends one extra renewal at start. A normal claim starts its watch within milliseconds with about 60 s left, so it sends no extra statement and adds no wait to a turn start. - No schema change. Mixed versions are safe: the renewal statement is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
11840f9904
commit
6917d0568d
4 files changed
+108
-12
No files matched your search
@@ -1202,6 +1202,52 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
|
||||
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id);
|
||||
});
|
||||
|
||||
it("starts each claimed run before claiming the next, within the agent's concurrency cap", async () => {
|
||||
// A claim commits a 60 s legacy controller lease that only the run's own
|
||||
// execution renews. A batch whose later claims are slow must not hold the
|
||||
// earlier claimed runs back from executing until their leases lapse.
|
||||
const [company] = await db.insert(companies).values({ name: "Claim batch", issuePrefix: "CB" }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Busy agent", role: "engineer", status: "idle",
|
||||
adapterType: "process", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 2 } } }).returning();
|
||||
const queued = [];
|
||||
for (let index = 0; index < 3; index++) {
|
||||
const [issue] = await db.insert(issues).values({ companyId: company.id, title: `Batch task ${index}`, status: "todo",
|
||||
assigneeAgentId: agent.id }).returning();
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment",
|
||||
status: "claimed", claimedAt: new Date() }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "queued",
|
||||
invocationSource: "assignment", wakeupRequestId: wake.id, responsibleUserId: "queue-test-user", contextSnapshot: { issueId: issue.id },
|
||||
createdAt: new Date(Date.now() - (3 - index) * 1_000) }).returning();
|
||||
await db.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() })
|
||||
.where(eq(issues.id, issue.id));
|
||||
queued.push(run);
|
||||
}
|
||||
const deps = callbacks(db);
|
||||
deps.getAgent.mockResolvedValue(agent);
|
||||
let releaseSecondClaim!: () => void;
|
||||
const secondClaimMayContinue = new Promise<void>(resolve => { releaseSecondClaim = resolve; });
|
||||
let budgetChecks = 0;
|
||||
deps.budgets.getInvocationBlock.mockImplementation(async () => {
|
||||
budgetChecks += 1;
|
||||
if (budgetChecks === 2) await secondClaimMayContinue;
|
||||
return null;
|
||||
});
|
||||
const admission = vi.fn(() => false);
|
||||
deps.isRunAdmissionClosed = admission;
|
||||
const pass = createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id);
|
||||
try {
|
||||
await vi.waitFor(() => expect(budgetChecks).toBe(2));
|
||||
// The first run is executing while the second claim is still in progress.
|
||||
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(queued[0]!.id);
|
||||
} finally { releaseSecondClaim(); }
|
||||
const claimed = await pass;
|
||||
expect(claimed.map(run => run.id)).toEqual([queued[0]!.id, queued[1]!.id]);
|
||||
expect(deps.executeRun.mock.calls.map(([runId]) => runId)).toEqual([queued[0]!.id, queued[1]!.id]);
|
||||
// The run-start admission gate is checked before every claim, and the cap holds.
|
||||
expect(admission.mock.calls.length).toBeGreaterThanOrEqual(3);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued[2]!.id)))[0].status).toBe("queued");
|
||||
});
|
||||
|
||||
it("leaves a queued run for the next server when shutdown closes admission during the queue scan", async () => {
|
||||
const { run, agent, wake } = await fixture("queued");
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
|
||||
|
||||
@@ -2589,17 +2589,21 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
noteDeclinedAgedClaim(queuedRun, err.message);
|
||||
continue;
|
||||
}
|
||||
if (claimed) {
|
||||
claimedRuns.push(claimed);
|
||||
dispatcher.claimed(queuedRun);
|
||||
declinedAgedClaimAt.delete(queuedRun.id);
|
||||
} else {
|
||||
if (!claimed) {
|
||||
noteDeclinedAgedClaim(queuedRun, "claim_declined");
|
||||
continue;
|
||||
}
|
||||
claimedRuns.push(claimed);
|
||||
dispatcher.claimed(queuedRun);
|
||||
declinedAgedClaimAt.delete(queuedRun.id);
|
||||
// Start each run as soon as it is claimed. The claim committed a legacy
|
||||
// controller lease that only the run's own execution renews; waiting
|
||||
// for the rest of a slow batch to be claimed let that lease lapse.
|
||||
startClaimedRun(claimed);
|
||||
}
|
||||
if (claimedRuns.length === 0) return [];
|
||||
return claimedRuns;
|
||||
|
||||
for (const claimedRun of claimedRuns) {
|
||||
function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) {
|
||||
const execution = executeRun(claimedRun.id).catch((err) => {
|
||||
logger.error(
|
||||
{ err, runId: claimedRun.id },
|
||||
@@ -2617,7 +2621,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
activeRunExecutionPromises.delete(execution);
|
||||
});
|
||||
}
|
||||
return claimedRuns;
|
||||
}).finally(async () => {
|
||||
// Dispatch promoted inputs after the agent start lock is released.
|
||||
try {
|
||||
|
||||
@@ -189,6 +189,42 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
expect(await renewLegacyControllerLease(db, run)).toBe(false);
|
||||
expect(await revokeExpiredLegacyController(db, run)).toBe(false);
|
||||
});
|
||||
it("renews at once when the watch starts on a lease older than one renewal interval", async () => {
|
||||
// The watch can start long after the claim (a slow claim batch). Waiting for
|
||||
// the first 10 s tick let a still-live lease reach its deadline unrenewed.
|
||||
const run = await seed();
|
||||
const [late] = await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp() + interval '1500 milliseconds'` })
|
||||
.where(eq(heartbeatRuns.id, run.id)).returning();
|
||||
const controller = new AbortController();
|
||||
const watch = watchLegacyControllerLease(db, late, controller);
|
||||
try {
|
||||
await new Promise(resolve => setTimeout(resolve, 2_000));
|
||||
expect(controller.signal.reason?.leaseLossTiming).toBeUndefined();
|
||||
expect(controller.signal.aborted).toBe(false);
|
||||
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
expect(saved.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now() + LEGACY_CONTROLLER_LEASE_MS - 5_000);
|
||||
expect(saved.controllerBootId).toBe(legacyControllerBootId);
|
||||
} finally { watch.stop(); }
|
||||
});
|
||||
it.each(["revoked by recovery", "reaped", "lapsed"] as const)(
|
||||
"a watch that starts late on a lease that was %s still loses it and never revives it", async outcome => {
|
||||
const run = await seed();
|
||||
await expire(run.id);
|
||||
if (outcome === "revoked by recovery") expect(await revokeExpiredLegacyController(db, run)).toBe(true);
|
||||
if (outcome === "reaped") await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
const controller = new AbortController();
|
||||
// The stale claim-time row still names this boot with a lease that has not expired locally.
|
||||
const watch = watchLegacyControllerLease(db, { ...run, controllerLeaseExpiresAt: new Date(Date.now() + 1_000) }, controller);
|
||||
try {
|
||||
await vi.waitFor(() => expect(controller.signal.aborted).toBe(true), { timeout: 2_000 });
|
||||
expect(controller.signal.reason).toBeInstanceOf(LegacyControllerLeaseLostError);
|
||||
expect(controller.signal.reason.leaseLossCause).toBe("renewal_rejected");
|
||||
const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
expect(after).toMatchObject({ status: before.status, controllerBootId: before.controllerBootId,
|
||||
controllerLeaseExpiresAt: before.controllerLeaseExpiresAt });
|
||||
} finally { watch.stop(); }
|
||||
});
|
||||
it("rejects dispatch at the lease deadline even if the database query never settles", async () => {
|
||||
const run = await seed();
|
||||
const hungDb = fakeRenewalDb(() => new Promise(() => {}));
|
||||
@@ -619,7 +655,9 @@ describe("legacy controller lease loss attribution", () => {
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
expect(controller.signal.reason.leaseLossCause).toBe("renewal_deadline");
|
||||
expect(controller.signal.reason.leaseLossTiming.deadlineLateMs).toBeGreaterThanOrEqual(60);
|
||||
expect(controller.signal.reason.leaseLossTiming.renewalPendingMs).toBeNull();
|
||||
// A watch that starts this close to the deadline renews at once; that renewal never answered.
|
||||
expect(controller.signal.reason.leaseLossTiming.renewalPendingMs).toBeGreaterThanOrEqual(60);
|
||||
expect(controller.signal.reason.leaseLossTiming.renewalCount).toBe(0);
|
||||
} finally { watch.stop(); }
|
||||
});
|
||||
});
|
||||
@@ -390,7 +390,8 @@ export function watchLegacyControllerLease(
|
||||
deadline = setTimeout(deadlinePassed, delayMs);
|
||||
deadline.unref();
|
||||
};
|
||||
armDeadline(Math.max(0, (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now()));
|
||||
const initialRemainingMs = (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now();
|
||||
armDeadline(Math.max(0, initialRemainingMs));
|
||||
const assertOwned = async (stage?: "dispatching") => {
|
||||
if (stopped) return;
|
||||
controller.signal.throwIfAborted();
|
||||
@@ -447,11 +448,19 @@ export function watchLegacyControllerLease(
|
||||
armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - renewedFrom)));
|
||||
}
|
||||
};
|
||||
const timer = setInterval(() => {
|
||||
const tick = () => {
|
||||
if (pending || stopped) return;
|
||||
pending = true;
|
||||
void assertOwned().catch(() => lost("renewal_error")).finally(() => { pending = false; });
|
||||
}, LEGACY_CONTROLLER_RENEW_MS);
|
||||
};
|
||||
const timer = setInterval(tick, LEGACY_CONTROLLER_RENEW_MS);
|
||||
timer.unref();
|
||||
// The watch can start well after the claim that wrote the lease (a slow
|
||||
// batch of claims, a loaded database). A lease older than one renewal
|
||||
// interval is renewed now rather than on the next tick. The renewal is the
|
||||
// same conditional statement as every other one: it only extends a lease
|
||||
// this boot still holds on a running row, so a lease that lapsed and was
|
||||
// revoked, reaped or taken over is still lost.
|
||||
if (initialRemainingMs < LEGACY_CONTROLLER_LEASE_MS - LEGACY_CONTROLLER_RENEW_MS) tick();
|
||||
return { assertOwned, stop() { stopped = true; clearInterval(timer); clearTimeout(deadline); } };
|
||||
}
|
||||
Reference in new issue
Block a user