mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
perf(recovery): keep a repeatedly deferring task's saved row; stop agent settlement from re-checking its backlog (#21905)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues check assigned tasks for work that has no
live run and no queued wake. Since #21872 they load only changed rows,
and they save pending deferrals so a replaced server keeps their
deadlines
> - Two costs remained. A running task's check defers to now + 30 s, and
each new deadline rewrote its saved deferral row: about two writes a
minute per running task per queue
> - Also, two writers that settle an agent after its runs end set the
agent's status and `updated_at` but not `last_heartbeat_at`. The
changed-rows agent rule then reads the write as a pause or resume and
re-checks all of that agent's tasks for about 5 minutes
> - This pull request keeps a repeatedly deferring task's saved row
until its deadline is well past, and makes both writers record a
heartbeat as `finalizeAgentStatus` does
> - The benefit is near-zero deferral writes for running tasks and no
whole-backlog re-check after an abandoned or stranded run settles an
agent
## Linked Issues or Issue Description
Refs #21872, #21861
**What happened?**
1. In `recovery/work.ts`, a deferring check upserted its
`issue_recovery_deferrals` row whenever the deadline changed. A deferral
to now + 30 s changes on every check, so a task deferring for 20 minutes
wrote its row 41 times. At 1,000 concurrent agents that is roughly 30 to
70 writes a second.
2. `execution-control-reconciliation.ts` (abandoned run past its
execution-control deadline) and `heartbeat/agent-status.ts`
`reconcileStrandedAgentStatuses` (agent left "running" with no live run)
set the agent to idle with `updated_at` but without `last_heartbeat_at`.
The agent clause of `changedTaskCondition` matches an agent that is not
running and whose `updated_at` is newer than its last heartbeat. So each
such write made the backstop re-check every task of that agent for about
5 minutes.
**Expected behavior**
A task that keeps deferring writes its row a small fixed number of
times. Settling an agent after its runs end does not re-check the
agent's other tasks; the run's own task is still checked promptly.
**Steps to reproduce**
1. Defer one task's check to now + 30 s on every check for 20 simulated
minutes and count inserts into `issue_recovery_deferrals`: 41.
2. Seed an agent left "running" with 140 settled reviews, settle it with
`reconcileAbandonedExecutionControl` or
`reconcileStrandedAgentStatuses`, and count stranded checks of those
reviews over the next backstop: 280 per agent.
**Paperclip version or commit**
`master` at b4fa1c7fb1.
## What Changed
- `recovery/work.ts`: a deferring check skips the upsert while this
server knows a saved row for the task whose deadline is less than
`DEFERRAL_REFRESH_MS` (5 minutes) past. It refreshes the row after that.
The refresh comes before the 10-minute stale prune margin, so a
still-deferring task's row never looks stale. A boot re-checks every
task with a row whatever its deadline, so the exact deadline only
matters to the conditional deletes and the stale prune, which still
match the value this server read or wrote. A cleared row, a failed write
or a reload still drop the server's memory of the row, so the next
deferral writes it again.
- `execution-control-reconciliation.ts` and `heartbeat/agent-status.ts`:
settling an agent after its runs end sets `lastHeartbeatAt` with
`updatedAt`, as `finalizeAgentStatus` already does, through a new
`settledHeartbeat(now)`. It records the settlement time only when the
agent already has a heartbeat; an agent with none keeps none, so the
first real heartbeat stays first (the existing "repairs a shutdown flag
without inventing a heartbeat" test still passes).
- `recovery/changed-tasks.ts`: the comment now states the contract:
every write that settles an agent's status after its runs end sets
`last_heartbeat_at`.
- Tests: a write-rate census in
`issue-recovery-saved-deferrals.test.ts`, and an agent-settle census and
guard in `issue-recovery-company-reload-census.test.ts`.
Why fix the writers, not the rule: the agent rule's discriminator
already is `last_heartbeat_at` against `updated_at` ("a run's finish
records a heartbeat"). Pauses, resumes, holds and configuration writes
leave `last_heartbeat_at` alone; run settlement records one. No other
column tells a settle-to-idle apart from a resume to idle, because both
end with status idle and a new `updated_at`. Since #21872, lifecycle
transitions also send their own agent hint, but configuration writes to
an idle agent still rely on this rule (the #21861 guard). So the sound
fix is for the two run-settling writers to follow the same contract as
`finalizeAgentStatus`. Recording the heartbeat at settlement has the
same effect on agent timers as a normal run finish.
## Verification
Red/green:
| Census | Before (master code) | After |
| --- | --- | --- |
| One task deferring every 30 s for 20 minutes: deferral writes | 41 (41
checks) | 4 (41 checks) |
| Same task: saved deadline behind now after 20 minutes | 0 (rewritten
each check) | 3 min (refreshed; the test fails above 6 min, and fails
with the refresh removed: 1 write, 19.5 min) |
| Two agents (140 settled reviews each) settled by an abandoned-run
reconcile and a stranded-status reconcile: backlog checks over the next
backstops | 560 | 0 |
| Same census: statements | 12,570 | 2,507 |
Guards:
- The abandoned run's task is checked within 1 minute of the reconcile
(passes before and after).
- The three deferral tests from #21872 still pass: replacement mid-wait
checks at the original deadline, the two-server interleaving keeps
server B's deadline, and the stale row of a task that left the queue is
removed.
Focused suites, each file run alone in the foreground:
`issue-recovery-company-reload-census.test.ts` 8 passed (38 s),
`issue-recovery-saved-deferrals.test.ts` 6 passed (14 s),
`heartbeat/agent-status.test.ts` and
`execution-control-reconciliation.test.ts` 60 passed,
`issue-recovery-idle-backstop.test.ts` 4 passed. An 18-file run earlier:
516 passed; the 4 failures were 1 test that this PR then fixed
(heartbeat for agents with none) and 3 in
`issue-recovery-idle-backstop.test.ts`, which pass alone (see Risks).
Server `tsc --noEmit`: passed. `pnpm verify`: not run; required CI
covers the broad suites.
## Risks
- A task that keeps deferring may have a saved deadline up to 5 minutes
old. Only the stale prune reads it, and it deletes a row only after the
task left the queue and the deadline is 10 minutes past, so a
still-deferring task's row is refreshed first.
- If another server deletes the row (its check stopped deferring and it
matched the same deadline) while this server keeps deferring, this
server rewrites the row at its next refresh (at most about 5 minutes)
instead of at its next check. Within that window a crash of this server
would leave the task to its successor's next changed-rows or full pass.
This needs the two servers to disagree about the same task, which only
happens through facts held in one server's memory.
- `last_heartbeat_at` now also moves when an abandoned or stranded run
settles the agent. Agent timers then count from that time, as they do
after a normal run finish.
- An agent that never recorded a heartbeat still has its whole backlog
re-checked once when such a writer settles it. This is rare: that agent
has not finished a run yet.
- Not from this PR: `issue-recovery-idle-backstop.test.ts` > "picks up a
finished run, a user message, a raised budget and a resumed agent within
a minute" hung once and failed 3 tests when run with other recovery
files in one process. It passes alone. Its waits have no time limits,
and its fixture agents can take on-demand wakes, which is the hang
pattern that #21867 removed from the agent-row census. It needs the same
treatment in a separate change.
- No schema change. Nothing on the run-start path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ac22f7f42d
commit
9de343b3fe
6 files changed
+123
-5
No files matched your search
@@ -11,6 +11,7 @@ import { documentService } from "../services/documents.js";
|
|||||||
import { issueService } from "../services/issues.js";
|
import { issueService } from "../services/issues.js";
|
||||||
import { createHeartbeatAgentStatus } from "../services/heartbeat/agent-status.js";
|
import { createHeartbeatAgentStatus } from "../services/heartbeat/agent-status.js";
|
||||||
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
|
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
|
||||||
|
import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js";
|
||||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||||
|
|
||||||
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; });
|
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; });
|
||||||
@@ -158,6 +159,67 @@ describeIssueRecovery("company hints and notification reloads (census)", () => {
|
|||||||
const checksOf = (events: Recorded[], issueId: string, after: number) => events.filter(event => event.topic === "stranded-assignments"
|
const checksOf = (events: Recorded[], issueId: string, after: number) => events.filter(event => event.topic === "stranded-assignments"
|
||||||
&& event.name === "check" && event.fields.issueId === issueId && event.at >= after);
|
&& event.name === "check" && event.fields.issueId === issueId && event.at >= after);
|
||||||
|
|
||||||
|
/** An agent left "running" with a large settled backlog: one abandoned
|
||||||
|
* run past its execution-control deadline, or (stranded) no run at all. */
|
||||||
|
async function settlingAgent(f: Awaited<ReturnType<typeof boot>>, name: string, abandoned: boolean) {
|
||||||
|
const old = new Date(Date.now() - 2 * HOUR);
|
||||||
|
const [agent] = await state.db.insert(agents).values({ companyId: f.companyId, name, role: "engineer", status: "running",
|
||||||
|
adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
|
||||||
|
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } },
|
||||||
|
createdAt: old, updatedAt: old, lastHeartbeatAt: old }).returning({ id: agents.id });
|
||||||
|
const tasks = await state.db.insert(issues).values(Array.from({ length: 140 }, (_, index) => ({ companyId: f.companyId,
|
||||||
|
title: `Settled ${index}`, status: "in_review", assigneeAgentId: agent!.id, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
|
||||||
|
let runTask: string | null = null;
|
||||||
|
if (abandoned) {
|
||||||
|
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Abandoned run's task", status: "in_progress",
|
||||||
|
assigneeAgentId: agent!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id });
|
||||||
|
runTask = task!.id;
|
||||||
|
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent!.id, status: "running",
|
||||||
|
startedAt: old, updatedAt: old, contextSnapshot: { issueId: runTask }, executionControlDeadlineAt: new Date(Date.now() - MINUTE) })
|
||||||
|
.returning({ id: heartbeatRuns.id });
|
||||||
|
await state.db.update(issues).set({ executionRunId: run!.id, updatedAt: old }).where(eq(issues.id, runTask));
|
||||||
|
}
|
||||||
|
return { agentId: agent!.id, tasks: tasks.map(row => row.id), runTask };
|
||||||
|
}
|
||||||
|
|
||||||
|
it("does not re-check an agent's settled backlog when an abandoned or stranded run settles the agent", async () => {
|
||||||
|
const f = await boot();
|
||||||
|
const abandoned = await bounded(settlingAgent(f, "Abandoned worker", true), () => "the abandoned agent to be seeded");
|
||||||
|
const stranded = await bounded(settlingAgent(f, "Stranded worker", false), () => "the stranded agent to be seeded");
|
||||||
|
await advance(MINUTE, f.tick);
|
||||||
|
f.events.length = 0;
|
||||||
|
const before = state.meter.read();
|
||||||
|
await bounded(reconcileAbandonedExecutionControl(state.db, new Date()), () => "the abandoned run to be reconciled");
|
||||||
|
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, f.tick);
|
||||||
|
await bounded(createHeartbeatAgentStatus(state.db).reconcileStrandedAgentStatuses(() => new Set(),
|
||||||
|
{ companyId: f.companyId, agentId: stranded.agentId }), () => "the stranded agent status to be reconciled");
|
||||||
|
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, f.tick);
|
||||||
|
const [abandonedAgent, strandedAgent] = await Promise.all([abandoned.agentId, stranded.agentId].map(id =>
|
||||||
|
state.db.select({ status: agents.status }).from(agents).where(eq(agents.id, id)).then(rows => rows[0]?.status)));
|
||||||
|
const backlog = new Set([...abandoned.tasks, ...stranded.tasks]);
|
||||||
|
const checks = f.events.filter(event => event.name === "check");
|
||||||
|
const result = { backlog: backlog.size, abandonedAgent, strandedAgent,
|
||||||
|
backlogChecks: checks.filter(event => backlog.has(String(event.fields.issueId))).length,
|
||||||
|
runTaskChecks: checks.filter(event => event.fields.issueId === abandoned.runTask).length,
|
||||||
|
byWake: byWake(f.events), statements: state.meter.read().statements - before.statements };
|
||||||
|
console.info(`agent-settle census: ${JSON.stringify(result)}`);
|
||||||
|
expect(abandonedAgent).not.toBe("running");
|
||||||
|
expect(strandedAgent).not.toBe("running");
|
||||||
|
expect(result.backlogChecks).toBe(0);
|
||||||
|
}, TEST_TIMEOUT_MS);
|
||||||
|
|
||||||
|
it("still checks an abandoned run's task promptly when the run is reconciled", async () => {
|
||||||
|
const f = await boot();
|
||||||
|
const abandoned = await bounded(settlingAgent(f, "Abandoned worker", true), () => "the abandoned agent to be seeded");
|
||||||
|
await advance(MINUTE, f.tick);
|
||||||
|
const reconciledAt = Date.now();
|
||||||
|
await bounded(reconcileAbandonedExecutionControl(state.db, new Date()), () => "the abandoned run to be reconciled");
|
||||||
|
await advance(2 * MINUTE, f.tick);
|
||||||
|
const [checked] = checksOf(f.events, abandoned.runTask!, reconciledAt);
|
||||||
|
expect(checked, "the abandoned run's task was not checked after reconciliation").toBeDefined();
|
||||||
|
expect(checked!.at - reconciledAt).toBeLessThanOrEqual(MINUTE);
|
||||||
|
}, TEST_TIMEOUT_MS);
|
||||||
|
|
||||||
it("does not re-check settled tasks for company hints, reloads or server replacements", async () => {
|
it("does not re-check settled tasks for company hints, reloads or server replacements", async () => {
|
||||||
const f = await boot();
|
const f = await boot();
|
||||||
const before = state.meter.read();
|
const before = state.meter.read();
|
||||||
|
|||||||
@@ -51,7 +51,8 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** One server: a coordinator and the recovery queues, with recorded checks. */
|
/** One server: a coordinator and the recovery queues, with recorded checks. */
|
||||||
async function server(behavior: { deferUntil: Map<string, number>; suppressed: () => boolean; checks: Array<{ id: string; at: number }> }) {
|
async function server(behavior: { deferUntil: Map<string, number>; suppressed: () => boolean; checks: Array<{ id: string; at: number }>;
|
||||||
|
rolling?: Set<string> }) {
|
||||||
const coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, canReconcile: () => true,
|
const coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, canReconcile: () => true,
|
||||||
onError: error => { throw error; } });
|
onError: error => { throw error; } });
|
||||||
running.add(coordinator);
|
running.add(coordinator);
|
||||||
@@ -65,6 +66,7 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => {
|
|||||||
behavior.checks.push({ id: scope.issueId, at: Date.now() });
|
behavior.checks.push({ id: scope.issueId, at: Date.now() });
|
||||||
const until = behavior.deferUntil.get(scope.issueId);
|
const until = behavior.deferUntil.get(scope.issueId);
|
||||||
if (until !== undefined && until > Date.now()) scope.onDeferredDeadline(until);
|
if (until !== undefined && until > Date.now()) scope.onDeferredDeadline(until);
|
||||||
|
if (behavior.rolling?.has(scope.issueId)) scope.onDeferredDeadline(Date.now() + 30_000);
|
||||||
};
|
};
|
||||||
const work = registerIssueRecoveryWork(state.db, coordinator, {
|
const work = registerIssueRecoveryWork(state.db, coordinator, {
|
||||||
stranded, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
stranded, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
||||||
@@ -180,6 +182,35 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => {
|
|||||||
expect(await bounded(saved(), () => "the deferral to be read")).toHaveLength(0);
|
expect(await bounded(saved(), () => "the deferral to be read")).toHaveLength(0);
|
||||||
}, TEST_TIMEOUT_MS);
|
}, TEST_TIMEOUT_MS);
|
||||||
|
|
||||||
|
it("writes a task's deferral a few times, not on every check, while it defers every 30 seconds", async () => {
|
||||||
|
const f = await bounded(seed(10), () => "the fixture to be seeded");
|
||||||
|
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||||
|
const target = f.tasks[2]!;
|
||||||
|
const behavior = { deferUntil: new Map<string, number>(), rolling: new Set([target]), suppressed: () => false,
|
||||||
|
checks: [] as Array<{ id: string; at: number }> };
|
||||||
|
const statements: string[] = [];
|
||||||
|
const priorDebug = state.db.$client.options.debug;
|
||||||
|
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
|
||||||
|
try {
|
||||||
|
await server(behavior);
|
||||||
|
await advance(20 * MINUTE);
|
||||||
|
} finally { state.db.$client.options.debug = priorDebug; }
|
||||||
|
const checks = behavior.checks.filter(check => check.id === target).length;
|
||||||
|
const writes = statements.filter(query => /insert into issue_recovery_deferrals/i.test(query)).length;
|
||||||
|
// The saved deadline still moves forward, so the row of a task that keeps
|
||||||
|
// deferring never looks stale to the prune (10 minutes past its deadline).
|
||||||
|
const [row] = await bounded(state.db.select({ dueAt: issueRecoveryDeferrals.dueAt }).from(issueRecoveryDeferrals)
|
||||||
|
.where(and(eq(issueRecoveryDeferrals.queue, "stranded-assignments"), eq(issueRecoveryDeferrals.issueId, target))),
|
||||||
|
() => "the deferral to be read");
|
||||||
|
const savedBehind = row ? Date.now() - row.dueAt.getTime() : null;
|
||||||
|
console.info(`rolling-deferral census: ${JSON.stringify({ minutes: 20, checks, deferralWrites: writes, savedBehindMs: savedBehind })}`);
|
||||||
|
expect(checks).toBeGreaterThanOrEqual(30);
|
||||||
|
expect(writes).toBeLessThanOrEqual(5);
|
||||||
|
expect(writes).toBeGreaterThan(1);
|
||||||
|
expect(savedBehind, "the saved deferral was not kept").not.toBeNull();
|
||||||
|
expect(savedBehind!).toBeLessThan(6 * MINUTE);
|
||||||
|
}, TEST_TIMEOUT_MS);
|
||||||
|
|
||||||
it("checks every task again after scheduling suppression, even with a recent saved full pass", async () => {
|
it("checks every task again after scheduling suppression, even with a recent saved full pass", async () => {
|
||||||
const f = await bounded(seed(30), () => "the fixture to be seeded");
|
const f = await bounded(seed(30), () => "the fixture to be seeded");
|
||||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
|||||||
import { randomUUID } from "node:crypto";
|
import { randomUUID } from "node:crypto";
|
||||||
import { logger } from "../middleware/logger.js";
|
import { logger } from "../middleware/logger.js";
|
||||||
import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm";
|
import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm";
|
||||||
|
import { settledHeartbeat } from "./heartbeat/agent-status.js";
|
||||||
import {
|
import {
|
||||||
agents,
|
agents,
|
||||||
heartbeatRuns,
|
heartbeatRuns,
|
||||||
@@ -292,7 +293,10 @@ export async function reconcileAbandonedExecutionControl(
|
|||||||
}
|
}
|
||||||
await tx
|
await tx
|
||||||
.update(agents)
|
.update(agents)
|
||||||
.set({ status: "idle", updatedAt: now })
|
// A run settling records a heartbeat, as finalizeAgentStatus does, so
|
||||||
|
// the changed-rows agent rule (recovery/changed-tasks.ts) does not
|
||||||
|
// read it as a pause or resume and re-check the agent's whole backlog.
|
||||||
|
.set({ status: "idle", lastHeartbeatAt: settledHeartbeat(now), updatedAt: now })
|
||||||
.where(
|
.where(
|
||||||
and(
|
and(
|
||||||
eq(agents.id, run.agentId),
|
eq(agents.id, run.agentId),
|
||||||
|
|||||||
@@ -75,6 +75,14 @@ export function settledAgentWorkCondition(liveRunIds: ReadonlySet<string>, now:
|
|||||||
OR c.receipt ? 'baseline' OR c.receipt->>'cleanupPending' = 'true'))`;
|
OR c.receipt ? 'baseline' OR c.receipt->>'cleanupPending' = 'true'))`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** The heartbeat a write records when it settles an agent after its runs end
|
||||||
|
* without finalizeAgentStatus: the settlement time, so the changed-rows agent
|
||||||
|
* rule does not re-check the agent's backlog, but never a first heartbeat for
|
||||||
|
* an agent that has none (that stays the first real one). */
|
||||||
|
export function settledHeartbeat(now: Date) {
|
||||||
|
return sql`case when ${agents.lastHeartbeatAt} is null then null else ${now.toISOString()}::timestamptz end`;
|
||||||
|
}
|
||||||
|
|
||||||
export function createHeartbeatAgentStatus(db: Db) {
|
export function createHeartbeatAgentStatus(db: Db) {
|
||||||
async function finalizeAgentStatus(
|
async function finalizeAgentStatus(
|
||||||
agentId: string,
|
agentId: string,
|
||||||
@@ -153,8 +161,10 @@ export function createHeartbeatAgentStatus(db: Db) {
|
|||||||
.orderBy(desc(heartbeatRuns.finishedAt), desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
.orderBy(desc(heartbeatRuns.finishedAt), desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||||
const nextStatus = settledStatus(lastRun?.status);
|
const nextStatus = settledStatus(lastRun?.status);
|
||||||
await notifyAttentionRetentionWork(tx, { companyId: candidate.companyId, sourceKind: "agent_error_alert" });
|
await notifyAttentionRetentionWork(tx, { companyId: candidate.companyId, sourceKind: "agent_error_alert" });
|
||||||
|
// Settling the agent's runs records a heartbeat, as finalizeAgentStatus
|
||||||
|
// does; see the agent rule in recovery/changed-tasks.ts.
|
||||||
const [agent] = await tx.update(agents).set({ status: nextStatus,
|
const [agent] = await tx.update(agents).set({ status: nextStatus,
|
||||||
errorReason: nextStatus === "error" ? errorReason(lastRun?.error) : null, updatedAt: now,
|
errorReason: nextStatus === "error" ? errorReason(lastRun?.error) : null, lastHeartbeatAt: settledHeartbeat(now), updatedAt: now,
|
||||||
}).where(and(eq(agents.id, candidate.id), eq(agents.companyId, candidate.companyId),
|
}).where(and(eq(agents.id, candidate.id), eq(agents.companyId, candidate.companyId),
|
||||||
eq(agents.status, "running"), eq(agents.lifecycleState, "ready"))).returning();
|
eq(agents.status, "running"), eq(agents.lifecycleState, "ready"))).returning();
|
||||||
await logActivity(tx as unknown as Db, {
|
await logActivity(tx as unknown as Db, {
|
||||||
|
|||||||
@@ -6,7 +6,11 @@ import { issues } from "@paperclipai/db";
|
|||||||
* tasks need recovery (a pause, resume, hold or configuration change). Run
|
* tasks need recovery (a pause, resume, hold or configuration change). Run
|
||||||
* lifecycle writes are left out: a run's finish records a heartbeat, and a run's
|
* lifecycle writes are left out: a run's finish records a heartbeat, and a run's
|
||||||
* start flips the agent to running, which makes none of its other tasks
|
* start flips the agent to running, which makes none of its other tasks
|
||||||
* stranded. Every run start would otherwise re-check all of the agent's tasks
|
* stranded. Every write that settles an agent's status after its runs end
|
||||||
|
* (finalizeAgentStatus, abandoned-run and stranded-status reconciliation)
|
||||||
|
* must set last_heartbeat_at with updated_at (see settledHeartbeat), or this
|
||||||
|
* clause re-checks all of the agent's tasks. An agent with no heartbeat yet
|
||||||
|
* keeps none, so its settlement still re-checks its (new) backlog once. Every run start would otherwise re-check all of the agent's tasks
|
||||||
* for two backstop windows. */
|
* for two backstop windows. */
|
||||||
export function changedTaskCondition(since: Date | SQL) {
|
export function changedTaskCondition(since: Date | SQL) {
|
||||||
const at = since instanceof Date ? sql`${since.toISOString()}::timestamptz` : since;
|
const at = since instanceof Date ? sql`${since.toISOString()}::timestamptz` : since;
|
||||||
|
|||||||
@@ -19,6 +19,11 @@ type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: numb
|
|||||||
const PAGE_SIZE = 100;
|
const PAGE_SIZE = 100;
|
||||||
/** How long past its deadline a saved deferral of a task that left its queue is kept. */
|
/** How long past its deadline a saved deferral of a task that left its queue is kept. */
|
||||||
const STALE_DEFERRAL_MS = 10 * 60_000;
|
const STALE_DEFERRAL_MS = 10 * 60_000;
|
||||||
|
/** How long past its saved deadline a task that keeps deferring leaves its row as is.
|
||||||
|
* A boot re-checks every task with a row whatever its deadline, so only the
|
||||||
|
* stale prune reads it; refreshing well before STALE_DEFERRAL_MS keeps the
|
||||||
|
* row of a task that is still deferring from ever looking stale. */
|
||||||
|
const DEFERRAL_REFRESH_MS = STALE_DEFERRAL_MS / 2;
|
||||||
|
|
||||||
export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
||||||
and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0)
|
and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0)
|
||||||
@@ -181,7 +186,9 @@ export function registerIssueRecoveryWork(
|
|||||||
async deferral(id: string, companyId: string, dueAt: number | null): Promise<boolean> {
|
async deferral(id: string, companyId: string, dueAt: number | null): Promise<boolean> {
|
||||||
const known = deferred.get(id);
|
const known = deferred.get(id);
|
||||||
if (dueAt !== null) {
|
if (dueAt !== null) {
|
||||||
if (known === dueAt) return true;
|
// A task that defers again and again (now + 30 s while a run lives)
|
||||||
|
// keeps its saved row until that row's deadline is well past.
|
||||||
|
if (known !== undefined && (known === dueAt || Date.now() < known + DEFERRAL_REFRESH_MS)) return true;
|
||||||
deferred.delete(id);
|
deferred.delete(id);
|
||||||
const saved = await db.execute(sql`insert into issue_recovery_deferrals (queue, issue_id, company_id, due_at)
|
const saved = await db.execute(sql`insert into issue_recovery_deferrals (queue, issue_id, company_id, due_at)
|
||||||
values (${queue}, ${id}::uuid, ${companyId}::uuid, ${at(dueAt)})
|
values (${queue}, ${id}::uuid, ${companyId}::uuid, ${at(dueAt)})
|
||||||
|
|||||||
Reference in new issue
Block a user