mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.
**Subsystem affected**
Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).
**Current behavior**
`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.
**Proposed behavior**
Design. For each task queue, a run event of run R on task T can change:
| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |
The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.
- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.
Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:
- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.
Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:
1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.
Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.
Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.
- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.
**Reason and benefit**
Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.
## What Changed
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.
## Verification
New tests. I saw each red test fail before the change.
- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).
Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).
| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |
New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.
With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).
## Risks
- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c1d3f90b37
commit
340157d1d6
30 files changed
+478
-139
No files matched your search
@@ -1,7 +1,10 @@
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
|
||||
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
|
||||
describeIssueRecovery("issue recovery production producers and assignments", () => {
|
||||
@@ -51,4 +54,55 @@ describeIssueRecovery("issue recovery production producers and assignments", ()
|
||||
await completed(issue.id); await finish(issue.id); await quiet();
|
||||
}, 90_000);
|
||||
|
||||
async function scopedStart() {
|
||||
const f = await fixture(); await start({}); await settled({ backstopPending: true });
|
||||
const task = async (status: "todo" | "in_progress" | "done") => (await state.db.insert(issues).values({ companyId: f.companyId,
|
||||
title: `Task ${status}`, status, assigneeAgentId: f.agentId }).returning())[0];
|
||||
const run = async (issueId: string, values: Partial<typeof heartbeatRuns.$inferInsert>) => (await state.db.insert(heartbeatRuns)
|
||||
.values({ companyId: f.companyId, agentId: f.agentId, contextSnapshot: { issueId }, ...values }).returning())[0];
|
||||
return { f, task, run };
|
||||
}
|
||||
async function recovered(issueId: string, sourceRunId: string) {
|
||||
await vi.waitFor(async () => {
|
||||
const runs = await state.db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`);
|
||||
const actions = await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
||||
expect(runs.filter(row => row.id !== sourceRunId).length + actions.length).toBeGreaterThan(0);
|
||||
}, { timeout: 15_000 });
|
||||
}
|
||||
it("restarts a task whose run failed from that run's own hint", async () => {
|
||||
const { f, task, run } = await scopedStart();
|
||||
const issue = await task("in_progress");
|
||||
const failed = await run(issue.id, { status: "failed", errorCode: "adapter_failed", finishedAt: new Date() });
|
||||
await state.db.transaction(tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId: failed.id, agentId: f.agentId }));
|
||||
await recovered(issue.id, failed.id); await finish(issue.id);
|
||||
}, 90_000);
|
||||
it("starts the agent's queued task when its run ends and frees the agent", async () => {
|
||||
const { f, task, run } = await scopedStart();
|
||||
const previous = await task("done");
|
||||
const ended = await run(previous.id, { status: "succeeded", finishedAt: new Date() });
|
||||
const next = await task("todo");
|
||||
await run(next.id, { status: "queued", invocationSource: "assignment", triggerDetail: "system" });
|
||||
await state.db.transaction(tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId: ended.id, agentId: f.agentId }));
|
||||
await completed(next.id); await finish(next.id);
|
||||
}, 90_000);
|
||||
it("restarts a task whose run the system cancelled", async () => {
|
||||
const { f, task, run } = await scopedStart();
|
||||
const issue = await task("in_progress");
|
||||
const cancelled = await run(issue.id, { status: "cancelled", finishedAt: new Date() });
|
||||
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: cancelled.id }));
|
||||
await recovered(issue.id, cancelled.id); await finish(issue.id);
|
||||
}, 90_000);
|
||||
it("restarts a task whose run vanished without a final event once its lease lapses", async () => {
|
||||
const f = await fixture();
|
||||
const [issue] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Lost run", status: "in_progress",
|
||||
assigneeAgentId: f.agentId }).returning();
|
||||
const [lost] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
|
||||
runtimeMode: "legacy", controllerBootId: randomUUID(), controllerLeaseExpiresAt: new Date(Date.now() + 5_000),
|
||||
contextSnapshot: { issueId: issue.id }, processPid: 2_000_000_000, startedAt: new Date() }).returning();
|
||||
await state.db.update(issues).set({ executionRunId: lost.id, checkoutRunId: lost.id }).where(eq(issues.id, issue.id));
|
||||
await start({});
|
||||
await recovered(issue.id, lost.id);
|
||||
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lost.id)))[0].status).not.toBe("running");
|
||||
await finish(issue.id);
|
||||
}, 90_000);
|
||||
});
|
||||
@@ -3,12 +3,18 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { agents, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db";
|
||||
import { agents, budgetPolicies, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
||||
import { registerIssueRecoveryWork } from "../services/recovery/work.js";
|
||||
import { notifyIssueRecovery } from "../services/issue-recovery-notifications.js";
|
||||
import { recoveryService } from "../services/recovery/service.js";
|
||||
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_MAX_ISSUE_IDS, notifyIssueRecovery, resolveIssueGraph }
|
||||
from "../services/issue-recovery-notifications.js";
|
||||
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
|
||||
import { notifyRunDispatch } from "../services/run-dispatch-notifications.js";
|
||||
import { promoteDeferredWorkIntents } from "../services/work-intent-outbox.js";
|
||||
import { taskWatchdogService } from "../services/task-watchdogs.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
@@ -49,11 +55,12 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
return query;
|
||||
}) as typeof state.db.select);
|
||||
await start(); await settled();
|
||||
expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(8);
|
||||
expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(11);
|
||||
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
|
||||
const [foreignSource] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning();
|
||||
const [foreignAction] = await state.db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id,
|
||||
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "foreign-query-budget", nextAction: "Inspect" }).returning();
|
||||
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId })); await settled();
|
||||
const before = { ...scans };
|
||||
for (let index = 0; index < 100; index++) await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
@@ -64,12 +71,8 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, foreignAction.id)))[0].status).toBe("active");
|
||||
expect(await originalSelect().from(heartbeatRuns)).toHaveLength(0); await quiet();
|
||||
});
|
||||
it("limits an ordinary comment to its issue among one thousand open assignments", async () => {
|
||||
const f = await fixture();
|
||||
const rows = await state.db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
|
||||
companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId,
|
||||
}))).returning({ id: issues.id });
|
||||
const checked: string[] = [];
|
||||
async function recordedRecovery(owner: Partial<Parameters<typeof registerIssueRecoveryWork>[2]>,
|
||||
options?: Parameters<typeof registerIssueRecoveryWork>[4]) {
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
|
||||
const register = state.coordinator.register.bind(state.coordinator);
|
||||
@@ -78,38 +81,181 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
|
||||
return pass;
|
||||
} });
|
||||
// Observe the actual paged reconstruction boundary without executing one
|
||||
// thousand adapters. The producer, transaction, coordinator and SQL are real;
|
||||
// the other acceptance cases exercise the resulting recovery decisions.
|
||||
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
|
||||
stranded: async scope => { checked.push(scope.issueId); },
|
||||
orphanBlockers: async () => {}, recoveryActions: async () => {},
|
||||
stranded: async () => {}, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
||||
dependencies: async () => ({ enqueueFailed: 0 }), locks: async () => {},
|
||||
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false,
|
||||
}, () => true);
|
||||
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false, ...owner,
|
||||
}, () => true, options);
|
||||
await state.worker.ready;
|
||||
async function settleWork() {
|
||||
for (let step = 0; step < 120; step++) {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
if (state.coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return;
|
||||
}
|
||||
throw new Error("Recovery did not settle its committed work within two virtual minutes");
|
||||
}
|
||||
async function settleWork(options: { backstopPending?: boolean } = {}) {
|
||||
for (let step = 0; step < 120; step++) {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
const next = state.coordinator.nextWakeAt();
|
||||
const settledNow = next === null || (options.backstopPending === true && next > Date.now() + 60_000);
|
||||
if (settledNow && idleWorkSnapshot().active === 0) return;
|
||||
}
|
||||
throw new Error("Recovery did not settle its committed work within two virtual minutes");
|
||||
}
|
||||
async function runGraph(companyId: string, agentId: string) {
|
||||
const [other] = await state.db.insert(agents).values({ companyId, name: "Other agent", role: "engineer", status: "idle",
|
||||
adapterType: "process", adapterConfig: {} }).returning({ id: agents.id });
|
||||
const [parent] = await state.db.insert(issues).values({ companyId, title: "Parent", status: "in_progress", assigneeAgentId: agentId })
|
||||
.returning({ id: issues.id });
|
||||
const [task] = await state.db.insert(issues).values({ companyId, title: "Run task", status: "in_progress", assigneeAgentId: agentId,
|
||||
parentId: parent.id }).returning({ id: issues.id });
|
||||
const [dependent] = await state.db.insert(issues).values({ companyId, title: "Waits on the run task", status: "todo",
|
||||
assigneeAgentId: other.id }).returning({ id: issues.id });
|
||||
const [sibling] = await state.db.insert(issues).values({ companyId, title: "Next task of the agent", status: "todo",
|
||||
assigneeAgentId: agentId }).returning({ id: issues.id });
|
||||
await state.db.insert(issueRelations).values({ companyId, issueId: task.id, relatedIssueId: dependent.id, type: "blocks" });
|
||||
const [run] = await state.db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: task.id } }).returning({ id: heartbeatRuns.id });
|
||||
const backlog = async (count: number) => (await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({
|
||||
companyId, title: `Backlog ${index}`, status: "todo", assigneeAgentId: other.id,
|
||||
}))).returning({ id: issues.id })).map(row => row.id);
|
||||
return { runId: run.id, taskId: task.id, siblingId: sibling.id, backlog, affected: [parent.id, task.id, dependent.id].sort() };
|
||||
}
|
||||
it("limits an ordinary comment to its issue among one thousand open assignments", async () => {
|
||||
const f = await fixture();
|
||||
const rows = await state.db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
|
||||
companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId,
|
||||
}))).returning({ id: issues.id });
|
||||
const checked: string[] = [];
|
||||
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
||||
await settleWork();
|
||||
expect(new Set(checked).size).toBe(1000);
|
||||
checked.length = 0;
|
||||
await issueService(state.db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" });
|
||||
await settleWork();
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(checked).toEqual([rows[0].id]);
|
||||
// Company eligibility changes still reconstruct all assignments. Narrowing
|
||||
// comment hints must not hide unrelated work after an operator resumes it.
|
||||
checked.length = 0;
|
||||
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
||||
await settleWork();
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(new Set(checked).size).toBe(1000);
|
||||
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
||||
await settleWork();
|
||||
await quiet();
|
||||
});
|
||||
it("re-checks only the tasks a run lifecycle event can affect, whatever the backlog", async () => {
|
||||
const f = await fixture();
|
||||
const graph = await runGraph(f.companyId, f.agentId);
|
||||
await graph.backlog(1000);
|
||||
const checked: string[] = [];
|
||||
const assigned: string[] = [];
|
||||
await recordedRecovery({
|
||||
stranded: async scope => { checked.push(scope.issueId); },
|
||||
assignedWork: {
|
||||
listSuspects: (scope, cursor, limit) => recoveryService(state.db, { enqueueWakeup: async () => null })
|
||||
.assignedWork.listSuspects(scope, cursor, limit),
|
||||
checkIssue: async scope => { assigned.push(scope.issueId); },
|
||||
},
|
||||
}, { backstopMs: 60 * 60_000 });
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(new Set(checked).size).toBe(1004);
|
||||
expect(checked).toContain(graph.siblingId);
|
||||
async function lifecycleEvent(producer: (tx: object) => Promise<unknown>, standalone?: () => Promise<unknown>) {
|
||||
checked.length = 0; assigned.length = 0;
|
||||
const statements: string[] = [], prior = state.db.$client.options.debug;
|
||||
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
|
||||
try {
|
||||
if (standalone) await standalone(); else await state.db.transaction(tx => producer(tx));
|
||||
await settleWork({ backstopPending: true });
|
||||
} finally { state.db.$client.options.debug = prior; }
|
||||
expect([...new Set(checked)].sort()).toEqual(graph.affected);
|
||||
expect(assigned.every(id => graph.affected.includes(id))).toBe(true);
|
||||
await vi.advanceTimersByTimeAsync(31_000);
|
||||
return statements.length;
|
||||
}
|
||||
const finished = () => (tx: object) => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId });
|
||||
const first = await lifecycleEvent(finished());
|
||||
await lifecycleEvent(tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId: graph.runId, agentId: f.agentId }));
|
||||
await lifecycleEvent(tx => notifyRunDispatch(tx, { companyId: f.companyId, issueId: graph.taskId }));
|
||||
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "lock_released", issueId: graph.taskId }));
|
||||
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "accounting_settled", runId: graph.runId }));
|
||||
const recoveryActions = issueRecoveryActionService(state.db);
|
||||
await lifecycleEvent(async () => {}, () => recoveryActions.upsertSourceScoped({ companyId: f.companyId, sourceIssueId: graph.taskId,
|
||||
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "run-failed",
|
||||
nextAction: "Choose the next step", evidence: {} }));
|
||||
await lifecycleEvent(async () => {}, () => recoveryActions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: graph.taskId,
|
||||
status: "resolved", outcome: "restored", resolutionNote: "Recovered" }));
|
||||
await graph.backlog(4000);
|
||||
expect(await lifecycleEvent(finished())).toBe(first);
|
||||
});
|
||||
it("resolves a run's task graph through index lookups, not the backlog", async () => {
|
||||
const f = await fixture();
|
||||
const graph = await runGraph(f.companyId, f.agentId);
|
||||
await graph.backlog(2000);
|
||||
await state.db.execute(sql`analyze issues`);
|
||||
await state.db.execute(sql`analyze heartbeat_runs`);
|
||||
const [read] = await state.db.transaction(async tx => {
|
||||
expect(await resolveIssueGraph(tx as unknown as typeof state.db, { companyId: f.companyId, runId: graph.runId })).toEqual(graph.affected);
|
||||
return tx.execute<{ rows: number }>(sql`select (coalesce(seq_tup_read, 0) + coalesce(idx_tup_fetch, 0))::int as rows
|
||||
from pg_stat_xact_user_tables where relname = 'issues'`);
|
||||
});
|
||||
expect(read.rows).toBeLessThan(50);
|
||||
});
|
||||
it("finds a stranded task outside the affected set within the backstop bound", async () => {
|
||||
const f = await fixture();
|
||||
const graph = await runGraph(f.companyId, f.agentId);
|
||||
await graph.backlog(50);
|
||||
const checkedAt = new Map<string, number>();
|
||||
await recordedRecovery({ stranded: async scope => { checkedAt.set(scope.issueId, Date.now()); } });
|
||||
await settleWork({ backstopPending: true });
|
||||
const fullPassAt = Date.now();
|
||||
const [unrelated] = await graph.backlog(1);
|
||||
checkedAt.clear();
|
||||
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(checkedAt.has(unrelated)).toBe(false);
|
||||
expect(state.coordinator.nextWakeAt()).toBeLessThanOrEqual(fullPassAt + ISSUE_RECOVERY_BACKSTOP_MS);
|
||||
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
||||
await settleWork();
|
||||
expect(checkedAt.get(unrelated)! - fullPassAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 5_000);
|
||||
checkedAt.clear();
|
||||
await vi.advanceTimersByTimeAsync(20 * 60_000);
|
||||
expect(checkedAt.size).toBe(0);
|
||||
const [later] = await graph.backlog(1);
|
||||
const eventAt = Date.now();
|
||||
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
||||
await settleWork();
|
||||
expect(checkedAt.get(later)! - eventAt).toBeLessThanOrEqual(5_000);
|
||||
await quiet();
|
||||
});
|
||||
it("keeps the company pass for hold releases that can admit any task", async () => {
|
||||
const f = await fixture();
|
||||
const graph = await runGraph(f.companyId, f.agentId);
|
||||
const backlog = await graph.backlog(150);
|
||||
const checked: string[] = [];
|
||||
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
||||
await settleWork({ backstopPending: true });
|
||||
const everything = new Set([...graph.affected, graph.siblingId, ...backlog]);
|
||||
await state.db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
|
||||
metric: "billed_cents", windowKind: "calendar_month_utc", amount: 1000, hardStopEnabled: true, isActive: true });
|
||||
for (const input of [{ trigger: "accounting_settled" as const, runId: graph.runId }, { trigger: "pause_lifted" as const }]) {
|
||||
checked.length = 0;
|
||||
await state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, ...input }));
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(new Set(checked)).toEqual(everything);
|
||||
await vi.advanceTimersByTimeAsync(31_000);
|
||||
}
|
||||
});
|
||||
it("falls back to the company pass when a run affects more tasks than a hint names", async () => {
|
||||
const f = await fixture();
|
||||
const graph = await runGraph(f.companyId, f.agentId);
|
||||
const backlog = await graph.backlog(200);
|
||||
const checked: string[] = [];
|
||||
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
||||
await settleWork({ backstopPending: true });
|
||||
const dependents = await graph.backlog(ISSUE_RECOVERY_MAX_ISSUE_IDS);
|
||||
await state.db.insert(issueRelations).values(dependents.map(id => ({ companyId: f.companyId, issueId: graph.taskId,
|
||||
relatedIssueId: id, type: "blocks" as const })));
|
||||
checked.length = 0;
|
||||
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(new Set(checked)).toEqual(new Set([...graph.affected, graph.siblingId, ...backlog, ...dependents]));
|
||||
});
|
||||
it("checks only blocked tasks whose blockers are done, however many still wait", async () => {
|
||||
const f = await fixture();
|
||||
const [openBlocker, doneBlocker] = await state.db.insert(issues).values([
|
||||
@@ -126,28 +272,7 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
{ companyId: f.companyId, issueId: doneBlocker.id, relatedIssueId: ready.id, type: "blocks" as const },
|
||||
]);
|
||||
const checked: string[] = [];
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
|
||||
const register = state.coordinator.register.bind(state.coordinator);
|
||||
state.coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
|
||||
const pass = task.run(signal); state.passes.add(pass);
|
||||
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
|
||||
return pass;
|
||||
} });
|
||||
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
|
||||
stranded: async () => {}, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
||||
dependencies: async scope => { checked.push(scope.issueId); return { enqueueFailed: 0 }; }, locks: async () => {},
|
||||
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false,
|
||||
}, () => true);
|
||||
await state.worker.ready;
|
||||
async function settleWork() {
|
||||
for (let step = 0; step < 120; step++) {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
if (state.coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return;
|
||||
}
|
||||
throw new Error("Recovery did not settle its committed work within two virtual minutes");
|
||||
}
|
||||
await recordedRecovery({ dependencies: async scope => { checked.push(scope.issueId); return { enqueueFailed: 0 }; } });
|
||||
await settleWork();
|
||||
expect(checked).toEqual([ready.id]);
|
||||
checked.length = 0;
|
||||
@@ -156,8 +281,10 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
expect(checked).toEqual([ready.id]);
|
||||
checked.length = 0;
|
||||
await issueService(state.db).update(openBlocker.id, { status: "done" });
|
||||
await settleWork();
|
||||
await settleWork({ backstopPending: true });
|
||||
expect(new Set(checked)).toEqual(new Set([ready.id, ...waiting.map(row => row.id)]));
|
||||
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
||||
await settleWork();
|
||||
await quiet();
|
||||
});
|
||||
it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => {
|
||||
|
||||
@@ -50,7 +50,7 @@ export function issueRecoveryFixture() {
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
|
||||
async function start() {
|
||||
async function start(options: { backstopMs?: number } = { backstopMs: 0 }) {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true,
|
||||
onError: error => errors.push(error) });
|
||||
tasks.clear();
|
||||
@@ -63,7 +63,7 @@ export function issueRecoveryFixture() {
|
||||
return work;
|
||||
} });
|
||||
};
|
||||
worker = heartbeat.registerIssueStateRecoveryWork(coordinator, () => !draining);
|
||||
worker = heartbeat.registerIssueStateRecoveryWork(coordinator, () => !draining, options);
|
||||
dispatch = heartbeat.registerRunDispatchWork(coordinator, () => !draining);
|
||||
cancellation = heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !draining);
|
||||
await Promise.all([worker.ready, dispatch.ready, cancellation.ready]);
|
||||
@@ -77,7 +77,7 @@ export function issueRecoveryFixture() {
|
||||
void pass.then(() => passes.delete(pass), () => passes.delete(pass));
|
||||
return pass;
|
||||
} });
|
||||
const work = other.registerIssueStateRecoveryWork(competing, () => true);
|
||||
const work = other.registerIssueStateRecoveryWork(competing, () => true, { backstopMs: 0 });
|
||||
extraStops.push(() => competing.stop()); extraDeadlines.push(competing.nextWakeAt); await work.ready;
|
||||
}
|
||||
async function fixture() {
|
||||
@@ -115,11 +115,12 @@ export function issueRecoveryFixture() {
|
||||
.where(eq(agents.id, f.agentId));
|
||||
return failures;
|
||||
}
|
||||
async function settled() {
|
||||
async function settled(options: { backstopPending?: boolean } = {}) {
|
||||
await vi.waitFor(async () => {
|
||||
await Promise.allSettled([...passes]);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
const wake = coordinator.nextWakeAt();
|
||||
const next = coordinator.nextWakeAt();
|
||||
const wake = options.backstopPending && next !== null && next > Date.now() + 60_000 ? null : next;
|
||||
const detail = wake === null ? "" : JSON.stringify({ now: Date.now(), tasks: await Promise.all([...tasks].map(async ([queue, task]) => ({ queue, at: task.nextRunAt?.(), pending: await task.hasPending() }))),
|
||||
runs: (await db.select().from(heartbeatRuns)).map(run => ({ id: run.id, status: run.status, error: run.errorCode, lease: run.controllerLeaseExpiresAt, reason: run.contextSnapshot?.wakeReason })) });
|
||||
expect(wake, detail).toBeNull();
|
||||
|
||||
@@ -104,7 +104,7 @@ describe("residual recovery production matrix", () => {
|
||||
const admitted = () => !draining && !standby;
|
||||
await heartbeat.registerResidualRecoveryWork(coordinator, admitted).ready;
|
||||
await heartbeat.registerActiveRunSilenceWork(coordinator, admitted).ready;
|
||||
await heartbeat.registerIssueStateRecoveryWork(coordinator, admitted).ready;
|
||||
await heartbeat.registerIssueStateRecoveryWork(coordinator, admitted, { backstopMs: 0 }).ready;
|
||||
await heartbeat.registerDeferredReceiptWork(coordinator, admitted).ready;
|
||||
await heartbeat.registerRunDispatchWork(coordinator, admitted).ready;
|
||||
await heartbeat.registerScheduledRetryCancellationWork(coordinator, admitted).ready;
|
||||
|
||||
@@ -697,3 +697,27 @@ it("drops an entry that vanished from a keyed scope and keeps other companies' s
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps a backstop hint's deadline without holding the worker busy", async () => {
|
||||
vi.useFakeTimers();
|
||||
const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) });
|
||||
coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError: vi.fn() });
|
||||
const backstopAt = Date.now() + 300_000;
|
||||
const load = vi.fn(async (_scope?: string) => []);
|
||||
const worker = registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.externalObjectRefresh, minimumPassIntervalMs: 0,
|
||||
decodeScope: hint => hint === "*" ? undefined : hint,
|
||||
backstop: scope => scope === "backstop" ? backstopAt : null,
|
||||
matches: () => true, load, check: async () => null,
|
||||
}, () => true);
|
||||
await worker.ready;
|
||||
load.mockClear();
|
||||
worker.invalidate("backstop");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(load).not.toHaveBeenCalled();
|
||||
expect(coordinator.nextWakeAt()).toBe(backstopAt);
|
||||
expect(worker.reconciliationSnapshot().reconciled).toBe(true);
|
||||
await vi.advanceTimersByTimeAsync(300_000);
|
||||
expect(load).toHaveBeenCalledWith("backstop", expect.anything());
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
});
|
||||
@@ -158,7 +158,7 @@ export async function wakeWorkspaceBusyWaitersInTx(tx: Db, input: {
|
||||
);
|
||||
const waiters = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(waiting).for("update");
|
||||
if (waiters.length === 0) return 0;
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.companyId, runId: input.runId });
|
||||
const woken = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ scheduledRetryAt: input.now, updatedAt: input.now })
|
||||
|
||||
@@ -732,7 +732,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
|
||||
await notifyRunDispatch(tx, { companyId });
|
||||
await notifyRunDispatch(tx, { companyId, issueId: issue.id });
|
||||
const queuedRun = await tx
|
||||
.insert(heartbeatRuns)
|
||||
.values({
|
||||
@@ -811,7 +811,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
|
||||
await notifyRunDispatch(tx, { companyId });
|
||||
await notifyRunDispatch(tx, { companyId, issueId: issue.id });
|
||||
const queuedRun = await tx
|
||||
.insert(heartbeatRuns)
|
||||
.values({
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
||||
import { assertAgentInteraction } from "./agent-interaction-policy.js";
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyIssueRecovery, notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, asc, desc, eq, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
@@ -290,7 +290,7 @@ export function documentService(db: Db) {
|
||||
.where(eq(issueDocuments.issueId, issue.id));
|
||||
const fallbackKey = nextAvailableDocumentKey(key, issueDocumentKeys.map((row) => row.key));
|
||||
|
||||
await notifyIssueRecovery(tx, { companyId: issue.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const [document] = await tx
|
||||
.insert(documents)
|
||||
.values({
|
||||
@@ -392,7 +392,7 @@ export function documentService(db: Db) {
|
||||
});
|
||||
}
|
||||
|
||||
await notifyIssueRecovery(tx, { companyId: issue.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const nextRevisionNumber = existing.latestRevisionNumber + 1;
|
||||
const [revision] = await tx
|
||||
.insert(documentRevisions)
|
||||
@@ -455,7 +455,7 @@ export function documentService(db: Db) {
|
||||
throw conflict("Document does not exist yet", { key });
|
||||
}
|
||||
|
||||
await notifyIssueRecovery(tx, { companyId: issue.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const [document] = await tx
|
||||
.insert(documents)
|
||||
.values({
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { notifyLiveRunRecovery, notifyLiveResourceRecovery } from "./live-run-recovery-notifications.js";
|
||||
import { notifyAttentionRetentionWork } from "./retention-work-notifications.js";
|
||||
import { notifyAttentionRetentionWork, notifyIssueAttentionRetentionWork } from "./retention-work-notifications.js";
|
||||
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
||||
import { notifyRunDispatch } from "./run-dispatch-notifications.js";
|
||||
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
||||
@@ -9,9 +9,14 @@ import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./deliv
|
||||
|
||||
export type ExecutionWorkScope = { companyId: string; agentId?: string; runId?: string; issueId?: string };
|
||||
|
||||
async function notifyExecutionAttentionRetention(transaction: object, scope: ExecutionWorkScope) {
|
||||
if (scope.issueId) await notifyIssueAttentionRetentionWork(transaction, { companyId: scope.companyId, issueId: scope.issueId });
|
||||
else await notifyAttentionRetentionWork(transaction, scope);
|
||||
}
|
||||
|
||||
export async function notifyExecutionRunWork(transaction: object, scope?: ExecutionWorkScope) {
|
||||
if (scope) {
|
||||
await notifyAttentionRetentionWork(transaction, scope);
|
||||
await notifyExecutionAttentionRetention(transaction, scope);
|
||||
await notifyRunDispatch(transaction, scope);
|
||||
await notifyStatusCardIssueChange(transaction, scope.companyId);
|
||||
await notifyDeferredExecutionEligibility(transaction, scope);
|
||||
@@ -36,7 +41,7 @@ export function writeExecutionControl<T>(
|
||||
await notifyRunDispatch(transaction, scope);
|
||||
if (scope) {
|
||||
await notifyStatusCardIssueChange(transaction, scope.companyId);
|
||||
await notifyAttentionRetentionWork(transaction, scope);
|
||||
await notifyExecutionAttentionRetention(transaction, scope);
|
||||
await notifyRoutineExecutionStatus(transaction, scope.issueId, scope.companyId);
|
||||
await notifyDeferredExecutionEligibility(transaction, scope);
|
||||
await notifyLiveRunRecovery(transaction, scope);
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyIssueRecovery, notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import type { ExternalConversationSettlementScope } from "./slack-conversation-lifecycle.js";
|
||||
|
||||
export async function notifyExternalConversationSettlement(tx: object, scope?: ExternalConversationSettlementScope) {
|
||||
await notifyIssueRecovery(tx, scope);
|
||||
if (scope?.issueId || scope?.runId) await notifyIssueGraphRecovery(tx, scope);
|
||||
else await notifyIssueRecovery(tx, scope);
|
||||
return notifyDeliveryWork(tx, DELIVERY_QUEUES.externalConversationSettlement,
|
||||
scope ? JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }) : "*");
|
||||
}
|
||||
@@ -4121,7 +4121,7 @@ export function heartbeatService(
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: null })
|
||||
.where(eq(heartbeatRuns.id, run.id)), { companyId: run.companyId });
|
||||
.where(eq(heartbeatRuns.id, run.id)), { companyId: run.companyId, runId: run.id });
|
||||
}
|
||||
nativeOwnershipHeld =
|
||||
nativeOwnershipHeld ||
|
||||
@@ -4725,7 +4725,8 @@ export function heartbeatService(
|
||||
schedulingSuppressed: async () => (await getSchedulingSuppression(true)).suppressed,
|
||||
prepareChatRetry,
|
||||
}, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerIssueStateRecoveryWork: (coordinator: Parameters<typeof registerIssueRecoveryWork>[1], canRun: () => boolean) =>
|
||||
registerIssueStateRecoveryWork: (coordinator: Parameters<typeof registerIssueRecoveryWork>[1], canRun: () => boolean,
|
||||
options?: Parameters<typeof registerIssueRecoveryWork>[4]) =>
|
||||
registerIssueRecoveryWork(db, coordinator, {
|
||||
stranded: scope => reconcileStrandedAssignedIssues({ ...scope, skipExternalConversationSettlement: true, skipRelatedRecovery: true }),
|
||||
orphanBlockers: recovery.reconcileUnassignedBlockingIssues,
|
||||
@@ -4739,7 +4740,7 @@ export function heartbeatService(
|
||||
checkIssue: recovery.assignedWork.checkIssue,
|
||||
},
|
||||
schedulingSuppressed: async () => (await getSchedulingSuppression(true)).suppressed,
|
||||
}, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
}, () => canRun() && readTaskDrain(new Date()) === null, options),
|
||||
registerRunDispatchWork: (coordinator: Parameters<typeof registerRunDispatchWork>[0], canRun: () => boolean) =>
|
||||
registerRunDispatchWork(coordinator, {
|
||||
promoteDueScheduledRetries,
|
||||
|
||||
@@ -5355,7 +5355,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
}
|
||||
|
||||
const queueOutcome = await db.transaction(async (tx) => {
|
||||
await notifyRunDispatch(tx, { companyId: agent.companyId });
|
||||
await notifyRunDispatch(tx, { companyId: agent.companyId, issueId: readNonEmptyString(enrichedContextSnapshot.issueId) });
|
||||
await tx.execute(
|
||||
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
|
||||
);
|
||||
|
||||
@@ -1210,7 +1210,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
}
|
||||
|
||||
await notifyAttentionRetentionWork(tx, { companyId: run.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId, runId: run.id });
|
||||
const wakeupRequest = await tx
|
||||
.insert(agentWakeupRequests)
|
||||
.values({
|
||||
@@ -2068,7 +2068,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
};
|
||||
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: scheduled.run.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: scheduled.run.companyId, runId: scheduled.run.id });
|
||||
const row = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
|
||||
@@ -1557,11 +1557,15 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
) {
|
||||
if (!wakeupRequestId) return;
|
||||
await db.transaction(async tx => {
|
||||
const [current] = await tx.select({ companyId: agentWakeupRequests.companyId, agentId: agentWakeupRequests.agentId, status: agentWakeupRequests.status })
|
||||
const [current] = await tx.select({ companyId: agentWakeupRequests.companyId, agentId: agentWakeupRequests.agentId, status: agentWakeupRequests.status,
|
||||
runId: agentWakeupRequests.runId, issueId: sql<string | null>`${agentWakeupRequests.payload}->>'issueId'` })
|
||||
.from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeupRequestId)).for("update");
|
||||
if (!current || current.status === "cancelled") return;
|
||||
if (current.status !== (patch?.status ?? status) || patch?.payload !== undefined || patch?.runId !== undefined) {
|
||||
await notifyExternalConversationSettlement(tx, { companyId: current.companyId });
|
||||
const runId = patch?.runId ?? current.runId;
|
||||
const issueId = current.issueId && /^[0-9a-f-]{36}$/i.test(current.issueId) ? current.issueId : undefined;
|
||||
await notifyExternalConversationSettlement(tx, issueId || runId
|
||||
? { companyId: current.companyId, issueId, runId: runId ?? undefined } : { companyId: current.companyId });
|
||||
}
|
||||
await notifyStrandedAgentStatus(tx, { companyId: current.companyId, agentId: current.agentId });
|
||||
await tx.update(agentWakeupRequests)
|
||||
@@ -2605,7 +2609,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
|
||||
await notifyRunDispatch(tx, { companyId: run.companyId });
|
||||
await notifyRunDispatch(tx, { companyId: run.companyId, issueId });
|
||||
const queuedRun = await tx
|
||||
.insert(heartbeatRuns)
|
||||
.values({
|
||||
|
||||
@@ -1054,7 +1054,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
), { companyId: run.companyId, runId: run.id });
|
||||
const workspaceFinalizeStatus = hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded";
|
||||
await recordWorkspaceFinalize(workspaceFinalizeStatus);
|
||||
if (adapterResult.nativeFinalization) {
|
||||
@@ -1135,7 +1135,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
), { companyId: run.companyId, runId: run.id });
|
||||
} catch (writeError) {
|
||||
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
|
||||
throw writeError;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { notifyAttentionRetentionWork } from "./retention-work-notifications.js";
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
@@ -345,7 +345,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
try {
|
||||
const created = await db.transaction(async (tx) => {
|
||||
await notifyAttentionRetentionWork(tx, { companyId: input.companyId });
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionDisposition);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
||||
@@ -417,7 +417,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
const [updated] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], tx => tx.update(issueRecoveryActions).set({ nextAction: input.nextAction, updatedAt: now })
|
||||
.where(and(eq(issueRecoveryActions.id, existing.id),
|
||||
inArray(issueRecoveryActions.status, [...ACTIVE_RECOVERY_ACTION_STATUSES]),
|
||||
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).returning(), { companyId: input.companyId });
|
||||
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).returning(), { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if (updated) return toReadModel(updated);
|
||||
}
|
||||
return existing;
|
||||
@@ -438,7 +438,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
Math.min(nextAttemptCount, effectiveMaxAttempts),
|
||||
);
|
||||
const [exhausted] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if ((!input.preserveExistingOwner && existing.cause !== input.cause)
|
||||
|| (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) {
|
||||
@@ -493,14 +493,14 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
),
|
||||
)
|
||||
.returning();
|
||||
}, { companyId: input.companyId });
|
||||
}, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if (!exhausted) {
|
||||
return retryUpsertSourceScoped(input, retryCount);
|
||||
}
|
||||
return toReadModel(exhausted);
|
||||
}
|
||||
const [updated] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if ((!input.preserveExistingOwner && existing.cause !== input.cause)
|
||||
|| (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) {
|
||||
@@ -566,7 +566,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
),
|
||||
)
|
||||
.returning();
|
||||
}, { companyId: input.companyId });
|
||||
}, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if (!updated) {
|
||||
return retryUpsertSourceScoped(input, retryCount);
|
||||
}
|
||||
@@ -575,14 +575,14 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
|
||||
try {
|
||||
const [created] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return tx
|
||||
.insert(issueRecoveryActions)
|
||||
.values(buildInsertValues(input, ownerType, now))
|
||||
.returning();
|
||||
}, { companyId: input.companyId });
|
||||
}, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return toReadModel(created!);
|
||||
} catch (error) {
|
||||
if (!isUniqueRecoveryActionConflict(error)) throw error;
|
||||
@@ -632,7 +632,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
isNotNull(nativeRunFinalizations.leaseOwner), inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted"]),
|
||||
)).limit(1);
|
||||
if (waiter) await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return tx
|
||||
@@ -646,7 +646,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
})
|
||||
.where(and(...predicates))
|
||||
.returning();
|
||||
}, { companyId: input.companyId });
|
||||
}, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
|
||||
return updated ? toReadModel(updated) : null;
|
||||
}
|
||||
|
||||
@@ -1,9 +1,20 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./delivery-work-notifications.js";
|
||||
|
||||
/** An issue hint is reserved for local facts such as comments. Lifecycle,
|
||||
* graph, run and eligibility changes retain company scope because their
|
||||
* consequences can include ancestors and shared execution references. */
|
||||
export type IssueRecoveryScope = { companyId: string; issueId?: string };
|
||||
export type IssueRecoveryScope = { companyId: string; issueId?: string; graphRunId?: string; graphIssueId?: string; backstop?: true;
|
||||
issueIds?: string[]; wholeCompany?: boolean };
|
||||
|
||||
export const ISSUE_RECOVERY_MAX_ISSUE_IDS = 100;
|
||||
export const ISSUE_RECOVERY_BACKSTOP_MS = 5 * 60_000;
|
||||
const UUID_PATTERN = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
|
||||
|
||||
const COMPANY_QUEUES: DeliveryQueue[] = [DELIVERY_QUEUES.orphanBlockers, DELIVERY_QUEUES.activeRecoveryActions, DELIVERY_QUEUES.taskWatchdogs];
|
||||
const TASK_QUEUES: DeliveryQueue[] = [DELIVERY_QUEUES.strandedAssignments, DELIVERY_QUEUES.assignedWorkInvariant,
|
||||
DELIVERY_QUEUES.resolvedDependencies, DELIVERY_QUEUES.staleIssueLocks];
|
||||
|
||||
export async function notifyIssueRecovery(transaction: object, scope?: IssueRecoveryScope) {
|
||||
const companyHint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*";
|
||||
@@ -14,3 +25,49 @@ export async function notifyIssueRecovery(transaction: object, scope?: IssueReco
|
||||
queue === DELIVERY_QUEUES.strandedAssignments || queue === DELIVERY_QUEUES.staleIssueLocks ? issueHint : companyHint);
|
||||
}
|
||||
}
|
||||
|
||||
export function decodeIssueRecoveryHint(hint: string): IssueRecoveryScope | undefined {
|
||||
return hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope;
|
||||
}
|
||||
|
||||
export function issueRecoveryScopeMatches(entry: { id: string; companyId: string }, scope?: IssueRecoveryScope) {
|
||||
if (!scope) return true;
|
||||
if (entry.companyId !== scope.companyId) return false;
|
||||
if (scope.issueId) return entry.id === scope.issueId;
|
||||
if (scope.graphRunId || scope.graphIssueId) return scope.wholeCompany === true || (scope.issueIds?.includes(entry.id) ?? false);
|
||||
return true;
|
||||
}
|
||||
|
||||
export async function notifyIssueGraphRecovery(transaction: object,
|
||||
scope: { companyId: string; runId?: string | null; issueId?: string | null }) {
|
||||
if (!scope.runId && !scope.issueId) return notifyIssueRecovery(transaction, { companyId: scope.companyId });
|
||||
const companyHint = JSON.stringify({ companyId: scope.companyId });
|
||||
for (const queue of COMPANY_QUEUES) await notifyDeliveryWork(transaction, queue, companyHint);
|
||||
const graphHint = JSON.stringify({ companyId: scope.companyId, graphRunId: scope.runId ?? undefined, graphIssueId: scope.issueId ?? undefined });
|
||||
for (const queue of TASK_QUEUES) {
|
||||
await notifyDeliveryWork(transaction, queue, graphHint);
|
||||
await notifyDeliveryWork(transaction, queue, JSON.stringify({ companyId: scope.companyId, backstop: true }));
|
||||
}
|
||||
}
|
||||
|
||||
export async function resolveIssueGraph(db: Db, scope: { companyId: string; runId?: string; issueId?: string }): Promise<string[] | null> {
|
||||
const company = sql`${scope.companyId}::uuid`;
|
||||
const run = scope.runId ? sql`${scope.runId}::uuid` : sql`null::uuid`;
|
||||
const rows = await db.execute<{ id: string }>(sql`with seed(id) as (
|
||||
select id from issues where company_id = ${company} and id = ${scope.issueId ?? null}::uuid
|
||||
union select issue_id from heartbeat_runs where company_id = ${company} and id = ${run} and issue_id is not null
|
||||
union select native_issue_id from heartbeat_runs where company_id = ${company} and id = ${run} and native_issue_id is not null
|
||||
union select issue.id from heartbeat_runs run
|
||||
cross join lateral (select coalesce(run.context_snapshot->>'issueId', run.context_snapshot->>'taskId') as saved) context
|
||||
inner join issues issue on issue.company_id = run.company_id
|
||||
and issue.id = case when context.saved ~* ${UUID_PATTERN} then context.saved::uuid end
|
||||
where run.company_id = ${company} and run.id = ${run}
|
||||
), affected(id) as (
|
||||
select id from seed
|
||||
union select parent_id from issues where company_id = ${company} and id in (select id from seed) and parent_id is not null
|
||||
union (select related_issue_id from issue_relations where company_id = ${company} and type = 'blocks'
|
||||
and issue_id in (select id from seed) limit ${ISSUE_RECOVERY_MAX_ISSUE_IDS + 1})
|
||||
) select id::text as id from affected limit ${ISSUE_RECOVERY_MAX_ISSUE_IDS + 1}`);
|
||||
const issueIds = rows.map(row => row.id).sort();
|
||||
return issueIds.length && issueIds.length <= ISSUE_RECOVERY_MAX_ISSUE_IDS ? issueIds : null;
|
||||
}
|
||||
@@ -12196,7 +12196,8 @@ export function issueService(db: Db) {
|
||||
|| patch.hiddenAt !== undefined || patch.conversationAgentId !== undefined || patch.parentId !== undefined
|
||||
|| patch.executionState !== undefined || patch.executionRunId !== undefined || patch.checkoutRunId !== undefined
|
||||
|| blockedByIssueIds !== undefined) {
|
||||
await notifyRunDispatch(tx, { companyId: receiptExisting.companyId });
|
||||
await notifyRunDispatch(tx, patch.parentId !== undefined && patch.parentId !== receiptExisting.parentId
|
||||
? { companyId: receiptExisting.companyId } : { companyId: receiptExisting.companyId, issueId: id });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: receiptExisting.companyId });
|
||||
await notifyActiveRunSilence(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
||||
@@ -12827,7 +12828,7 @@ export function issueService(db: Db) {
|
||||
await notifyStatusCardIssueChange(tx, issueCompany.companyId);
|
||||
await notifyIssueAttentionRetentionWork(tx, { companyId: issueCompany.companyId, issueId: id });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
||||
await notifyRunDispatch(tx, { companyId: issueCompany.companyId });
|
||||
await notifyRunDispatch(tx, { companyId: issueCompany.companyId, issueId: id });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: issueCompany.companyId });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
||||
await notifyRoutineExecutionStatus(tx, id);
|
||||
@@ -13238,7 +13239,7 @@ export function issueService(db: Db) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id });
|
||||
await notifyIssueAttentionRetentionWork(tx, { companyId: existing.companyId, issueId: id });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
||||
await notifyRunDispatch(tx, { companyId: existing.companyId });
|
||||
await notifyRunDispatch(tx, { companyId: existing.companyId, issueId: id });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: existing.companyId });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
||||
await notifyRoutineExecutionStatus(tx, id);
|
||||
@@ -13303,7 +13304,7 @@ export function issueService(db: Db) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: existing.companyId, issueId: id });
|
||||
}
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
||||
await notifyRunDispatch(tx, { companyId: existing.companyId });
|
||||
await notifyRunDispatch(tx, { companyId: existing.companyId, issueId: id });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: existing.companyId });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
||||
await notifyRoutineExecutionStatus(tx, id);
|
||||
|
||||
@@ -158,7 +158,7 @@ export async function terminalizeLegacyExecution(input: {
|
||||
};
|
||||
if (issueId) await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: current.companyId, issueId });
|
||||
await notifyInstructionCollection(tx, { companyId: current.companyId, runId: current.id });
|
||||
await notifyExecutionRunWork(tx, { companyId: current.companyId });
|
||||
await notifyExecutionRunWork(tx, { companyId: current.companyId, runId: current.id });
|
||||
let [updated] = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
|
||||
@@ -52,7 +52,7 @@ export async function materializeNativeProviderCapacityRetry(input: {
|
||||
executionRetryAccounting: accountingForScheduledRetry(run, NATIVE_PROVIDER_OVERLOADED_CODE, attempt),
|
||||
};
|
||||
await notifyAttentionRetentionWork(input.tx, { companyId: input.companyId });
|
||||
await notifyScheduledRetryDispatch(input.tx, { companyId: run.companyId });
|
||||
await notifyScheduledRetryDispatch(input.tx, { companyId: run.companyId, runId: run.id });
|
||||
const [wake] = await input.tx.insert(agentWakeupRequests).values({
|
||||
companyId: run.companyId, agentId: run.agentId, source: "automation", triggerDetail: "system",
|
||||
reason: NATIVE_PROVIDER_OVERLOADED_CODE, status: "queued", runId: retryId,
|
||||
|
||||
@@ -775,8 +775,8 @@ export async function reconcileSafeNativeReplacements(
|
||||
idempotencyKey: `native-safe-replacement:${run.id}`,
|
||||
})
|
||||
.returning();
|
||||
await notifyExecutionRunWork(tx, { companyId: run.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId });
|
||||
await notifyExecutionRunWork(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx.insert(heartbeatRuns).values({
|
||||
id: successorRunId,
|
||||
companyId: run.companyId,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
||||
import { notifyIssueAttentionRetentionWork } from "./retention-work-notifications.js";
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string };
|
||||
@@ -8,7 +8,7 @@ export type NativeStatusReassessmentScope = { companyId: string; issueId?: strin
|
||||
export async function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) {
|
||||
await notifyStatusCardIssueChange(tx, scope.companyId);
|
||||
await notifyIssueAttentionRetentionWork(tx, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId });
|
||||
await notifyIssueRecovery(tx, scope ? { companyId: scope.companyId } : undefined);
|
||||
await notifyIssueGraphRecovery(tx, scope);
|
||||
return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId }));
|
||||
}
|
||||
@@ -505,7 +505,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
|
||||
order by lr.created_at desc limit 1), false)`;
|
||||
}
|
||||
|
||||
async function listSuspects(scope: { companyId?: string; issueId?: string; createdAfter?: Date | null }, cursor: string | null, limit: number) {
|
||||
async function listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[]; createdAfter?: Date | null },
|
||||
cursor: string | null, limit: number) {
|
||||
const at = now();
|
||||
const liveRun = sql`exists (select 1 from heartbeat_runs r where r.company_id = ${issues.companyId}
|
||||
and (r.context_snapshot->>'issueId') = ${issues.id}::text
|
||||
@@ -521,6 +522,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
|
||||
return db.select({ id: issues.id, companyId: issues.companyId, updatedAt: issues.updatedAt }).from(issues).where(and(
|
||||
scope.companyId ? eq(issues.companyId, scope.companyId) : undefined,
|
||||
scope.issueId ? eq(issues.id, scope.issueId) : undefined,
|
||||
scope.issueIds ? inArray(issues.id, scope.issueIds) : undefined,
|
||||
cursor ? sql`${issues.id} > ${cursor}` : undefined,
|
||||
scope.createdAfter ? gte(issues.createdAt, scope.createdAfter) : undefined,
|
||||
inArray(issues.status, [...OPEN_STATUSES]),
|
||||
|
||||
@@ -2832,7 +2832,7 @@ export function recoveryService(
|
||||
const now = new Date();
|
||||
const retryAt = readProviderQuotaRetryAt(input.latestRun, now);
|
||||
return database.transaction(async (tx) => {
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.issue.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.issue.companyId, issueId: input.issue.id });
|
||||
const wakeup = await tx
|
||||
.insert(agentWakeupRequests)
|
||||
.values({
|
||||
@@ -3429,7 +3429,7 @@ export function recoveryService(
|
||||
created = Boolean(enqueuedRun);
|
||||
} else {
|
||||
scheduledRun = await db.transaction(async (tx) => {
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.issue.companyId });
|
||||
await notifyScheduledRetryDispatch(tx, { companyId: input.issue.companyId, issueId: input.issue.id });
|
||||
const wakeup = await tx
|
||||
.insert(agentWakeupRequests)
|
||||
.values({
|
||||
|
||||
@@ -3,7 +3,8 @@ import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql, type SQL } from
|
||||
import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js";
|
||||
import type { IssueRecoveryScope } from "../issue-recovery-notifications.js";
|
||||
import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope }
|
||||
from "../issue-recovery-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { recoveryActionSweepCondition } from "./blocker-stall-wakeups.js";
|
||||
|
||||
@@ -31,29 +32,69 @@ export function registerIssueRecoveryWork(
|
||||
schedulingSuppressed(): Promise<boolean>;
|
||||
/** The assigned-work invariant (assigned-work-sweep.ts). */
|
||||
assignedWork?: {
|
||||
listSuspects(scope: { companyId?: string; issueId?: string }, cursor: string | null, limit: number):
|
||||
listSuspects(scope: { companyId?: string; issueId?: string; issueIds?: string[] }, cursor: string | null, limit: number):
|
||||
Promise<Array<{ id: string; companyId: string; updatedAt: Date }>>;
|
||||
checkIssue(scope: Scope): Promise<unknown>;
|
||||
};
|
||||
},
|
||||
canRun: () => boolean,
|
||||
options: { backstopMs?: number } = {},
|
||||
) {
|
||||
function fullPasses() {
|
||||
const completedAt = new Map<string, number>();
|
||||
return {
|
||||
backstop: (scope: IssueRecoveryScope | undefined) => scope?.backstop
|
||||
? Math.max(completedAt.get("*") ?? 0, completedAt.get(scope.companyId) ?? 0) + (options.backstopMs ?? ISSUE_RECOVERY_BACKSTOP_MS)
|
||||
: null,
|
||||
loaded(scope: IssueRecoveryScope | undefined, complete: boolean) {
|
||||
if (complete && !scope?.issueId && (!isGraph(scope) || scope?.wholeCompany)) completedAt.set(scope?.companyId ?? "*", Date.now());
|
||||
},
|
||||
};
|
||||
}
|
||||
const reconstructionIntervalMs = (scope: IssueRecoveryScope | undefined) => scope?.backstop || scope?.graphRunId || scope?.graphIssueId ? 0 : 30_000;
|
||||
const scopeKey = (scope: IssueRecoveryScope | undefined) => scope
|
||||
? JSON.stringify([scope.companyId, scope.issueId, scope.graphRunId, scope.graphIssueId, scope.backstop]) : "*";
|
||||
const isGraph = (scope: IssueRecoveryScope | undefined) => Boolean(scope?.graphRunId || scope?.graphIssueId);
|
||||
const graphs = new Map<string, { at: number; ids: Promise<string[] | null> }>();
|
||||
async function resolveGraph(scope: IssueRecoveryScope | undefined) {
|
||||
if (!scope || !isGraph(scope)) return;
|
||||
const key = scopeKey(scope);
|
||||
for (const [other, cached] of graphs) if (cached.at < Date.now() - 2_000) graphs.delete(other);
|
||||
let cached = graphs.get(key);
|
||||
if (!cached) {
|
||||
cached = { at: Date.now(), ids: resolveIssueGraph(db, { companyId: scope.companyId, runId: scope.graphRunId, issueId: scope.graphIssueId }) };
|
||||
graphs.set(key, cached);
|
||||
}
|
||||
const ids = await cached.ids.catch(error => { graphs.delete(key); throw error; });
|
||||
scope.issueIds = ids ?? undefined;
|
||||
scope.wholeCompany = ids === null;
|
||||
}
|
||||
async function toCompany(scope: IssueRecoveryScope | undefined, worker: { invalidate(hint?: string): void }) {
|
||||
await resolveGraph(scope);
|
||||
if (!scope?.wholeCompany) return false;
|
||||
worker.invalidate(JSON.stringify({ companyId: scope.companyId }));
|
||||
return true;
|
||||
}
|
||||
function register(queue: DeliveryQueue, statuses: string[] | null, check: (scope: Scope) => Promise<unknown>, eligible?: SQL) {
|
||||
const cursors = new Map<string, string>();
|
||||
const passes = fullPasses();
|
||||
let enabled = false;
|
||||
return registerScopedDeadlineWork<Entry, IssueRecoveryScope>(coordinator, {
|
||||
const worker: ReturnType<typeof registerScopedDeadlineWork<Entry, IssueRecoveryScope>> = registerScopedDeadlineWork<Entry, IssueRecoveryScope>(coordinator, {
|
||||
queue,
|
||||
reconstructionIntervalMs: 30_000,
|
||||
reconstructionIntervalMs,
|
||||
backstop: passes.backstop,
|
||||
beforePass: async () => { enabled = !(await owner.schedulingSuppressed()); },
|
||||
decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope,
|
||||
matches: (entry, scope) => !scope || (entry.companyId === scope.companyId && (!scope.issueId || entry.id === scope.issueId)),
|
||||
decodeScope: decodeIssueRecoveryHint,
|
||||
matches: issueRecoveryScopeMatches,
|
||||
load: async scope => {
|
||||
const key = scope ? JSON.stringify([scope.companyId, scope.issueId]) : "*";
|
||||
const key = scopeKey(scope);
|
||||
if (!enabled) { cursors.delete(key); return []; }
|
||||
if (await toCompany(scope, worker)) return { rows: null, complete: true };
|
||||
const cursor = cursors.get(key);
|
||||
const rows = await db.select({ id: issues.id, companyId: issues.companyId }).from(issues).where(and(
|
||||
scope ? eq(issues.companyId, scope.companyId) : undefined,
|
||||
scope?.issueId ? eq(issues.id, scope.issueId) : undefined,
|
||||
scope?.issueIds ? inArray(issues.id, scope.issueIds) : undefined,
|
||||
cursor ? gt(issues.id, cursor) : undefined,
|
||||
statuses ? inArray(issues.status, statuses) : or(isNotNull(issues.checkoutRunId), isNotNull(issues.executionRunId)),
|
||||
statuses ? isNull(issues.hiddenAt) : undefined,
|
||||
@@ -63,6 +104,7 @@ export function registerIssueRecoveryWork(
|
||||
)).orderBy(asc(issues.id)).limit(PAGE_SIZE);
|
||||
const complete = rows.length < PAGE_SIZE;
|
||||
if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id);
|
||||
passes.loaded(scope, complete);
|
||||
return { rows: rows.map(row => ({ ...row, dueAt: Date.now() })), complete };
|
||||
},
|
||||
check: async entry => {
|
||||
@@ -73,6 +115,7 @@ export function registerIssueRecoveryWork(
|
||||
return next === null ? null : { ...entry, dueAt: next };
|
||||
},
|
||||
}, canRun);
|
||||
return worker;
|
||||
}
|
||||
/** Event-driven: run terminal states, hold settlements and boot mark the
|
||||
* company dirty; checks schedule their own grace and back-off deadlines.
|
||||
@@ -80,22 +123,27 @@ export function registerIssueRecoveryWork(
|
||||
function registerAssignedWork(sweep: NonNullable<typeof owner.assignedWork>) {
|
||||
type Suspect = Entry & { updatedAt: number };
|
||||
const cursors = new Map<string, string>();
|
||||
const passes = fullPasses();
|
||||
let enabled = false;
|
||||
return registerScopedDeadlineWork<Suspect, IssueRecoveryScope>(coordinator, {
|
||||
const worker: ReturnType<typeof registerScopedDeadlineWork<Suspect, IssueRecoveryScope>> = registerScopedDeadlineWork<Suspect, IssueRecoveryScope>(coordinator, {
|
||||
queue: DELIVERY_QUEUES.assignedWorkInvariant,
|
||||
reconstructionIntervalMs: 30_000,
|
||||
reconstructionIntervalMs,
|
||||
backstop: passes.backstop,
|
||||
batchSize: 25,
|
||||
beforePass: async () => { enabled = !(await owner.schedulingSuppressed()); },
|
||||
decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope,
|
||||
matches: (entry, scope) => !scope || (entry.companyId === scope.companyId && (!scope.issueId || entry.id === scope.issueId)),
|
||||
decodeScope: decodeIssueRecoveryHint,
|
||||
matches: issueRecoveryScopeMatches,
|
||||
priority: entry => entry.updatedAt,
|
||||
mergeEntry: (pending, loaded) => ({ ...loaded, dueAt: Math.max(pending.dueAt, loaded.dueAt) }),
|
||||
load: async scope => {
|
||||
const key = scope ? JSON.stringify([scope.companyId, scope.issueId]) : "*";
|
||||
const key = scopeKey(scope);
|
||||
if (!enabled) { cursors.delete(key); return []; }
|
||||
const rows = await sweep.listSuspects(scope ?? {}, cursors.get(key) ?? null, PAGE_SIZE);
|
||||
if (await toCompany(scope, worker)) return { rows: null, complete: true };
|
||||
const rows = await sweep.listSuspects(scope ? { companyId: scope.companyId, issueId: scope.issueId,
|
||||
issueIds: scope.issueIds } : {}, cursors.get(key) ?? null, PAGE_SIZE);
|
||||
const complete = rows.length < PAGE_SIZE;
|
||||
if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id);
|
||||
passes.loaded(scope, complete);
|
||||
return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, updatedAt: row.updatedAt.getTime(), dueAt: Date.now() })), complete };
|
||||
},
|
||||
check: async entry => {
|
||||
@@ -106,6 +154,7 @@ export function registerIssueRecoveryWork(
|
||||
return next === null ? null : { ...entry, dueAt: next };
|
||||
},
|
||||
}, canRun);
|
||||
return worker;
|
||||
}
|
||||
function registerCompany(queue: DeliveryQueue, kind: "blockers" | "actions", check: (scope: { companyId: string }) => Promise<number | null | void>) {
|
||||
const cursors = new Map<string, string>();
|
||||
|
||||
@@ -166,8 +166,8 @@ const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'nativ
|
||||
/** A run's accounting settled: an admission hold may have cleared. Held work
|
||||
* (queued runs, saved messages and Retries, and tasks whose start or
|
||||
* continuation was refused) waits passively, so wake its owners. */
|
||||
async function notifyAccountingHoldRelease(tx: object, companyId: string) {
|
||||
await promoteDeferredWorkIntents(tx, { companyId, trigger: "accounting_settled" });
|
||||
async function notifyAccountingHoldRelease(tx: object, companyId: string, runId: string) {
|
||||
await promoteDeferredWorkIntents(tx, { companyId, trigger: "accounting_settled", runId });
|
||||
}
|
||||
|
||||
export const ACCOUNTING_CLOSURE_TIMEOUT_MS = REMOTE_RUNNER_RECONNECT_GRACE_MS + 3 * 60_000;
|
||||
@@ -289,11 +289,11 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
if (run.costAccountedAt) {
|
||||
if (object(run.usageJson).accountingLateReceiptPending === true) {
|
||||
// Clearing late debt can release an admission hold too.
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await notifyAccountingHoldRelease(tx, run.companyId, run.id);
|
||||
return reconcileLateRunUsageReceiptInTransaction(tx, run, publications);
|
||||
}
|
||||
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await notifyAccountingHoldRelease(tx, run.companyId, run.id);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, run.id));
|
||||
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
|
||||
projectId: text(object(object(run.usageJson).ledgerScope).projectId) });
|
||||
@@ -333,7 +333,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
// Acknowledgement is part of the same transaction. Budget checks inside
|
||||
// this transaction must not mistake this receipt for missing accounting.
|
||||
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await notifyAccountingHoldRelease(tx, run.companyId, run.id);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: false, costAccountedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
|
||||
await tx.update(budgetReservations).set({ state: preProviderFailure ? "released" : "settled", settledAt: new Date() })
|
||||
.where(and(eq(budgetReservations.companyId, run.companyId), eq(budgetReservations.runId, run.id), eq(budgetReservations.state, "held")));
|
||||
@@ -426,7 +426,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId, projectId: text(scope.projectId) });
|
||||
// Late usage reported by a later run can arrive before this charge posts.
|
||||
const lateDebt = usage.accountingLateReceiptPending === true;
|
||||
if (!lateDebt) await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
if (!lateDebt) await notifyAccountingHoldRelease(tx, run.companyId, run.id);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: lateDebt, costAccountedAt: new Date(), accountingProjectionVersion: "v2",
|
||||
accountingLastError: lateDebt ? "late_usage_requires_reconciliation" : null, accountingLastAttemptAt: new Date(),
|
||||
// A complete receipt arriving later may price this charge (see usage-receipts).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { notifyIssueRecovery, notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type RunDispatchScope = { companyId: string };
|
||||
export type RunDispatchScope = { companyId: string; issueId?: string | null; runId?: string | null };
|
||||
|
||||
export async function notifyQueuedRunDispatch(transaction: object, scope?: RunDispatchScope) {
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.queuedRunDispatch,
|
||||
@@ -9,7 +9,8 @@ export async function notifyQueuedRunDispatch(transaction: object, scope?: RunDi
|
||||
}
|
||||
|
||||
export async function notifyRunDispatch(transaction: object, scope?: RunDispatchScope) {
|
||||
await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined);
|
||||
if (scope?.issueId || scope?.runId) await notifyIssueGraphRecovery(transaction, scope);
|
||||
else await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined);
|
||||
const hint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*";
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativeProcessSessions, hint);
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement, hint);
|
||||
|
||||
@@ -26,6 +26,7 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
admitDueWhileActive?: boolean;
|
||||
};
|
||||
reconstructionIntervalMs?: number | ((scope: Scope | undefined) => number);
|
||||
backstop?(scope: Scope | undefined): number | null;
|
||||
beforePass?(): Promise<void>;
|
||||
decodeScope(hint: string): Scope | undefined;
|
||||
matches(entry: Entry, scope?: Scope): boolean;
|
||||
@@ -67,6 +68,12 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
decodedScopes.set(hint, scope);
|
||||
return scope;
|
||||
}
|
||||
function backstopAt(hint: string) {
|
||||
if (!options.backstop) return null;
|
||||
try { return options.backstop(decode(hint)); } catch { return null; }
|
||||
}
|
||||
const holdsIdle = (hint: string) => backstopAt(hint) === null;
|
||||
const gate = (hint: string) => Math.max(reconstructAfter.get(hint) ?? 0, backstopAt(hint) ?? 0);
|
||||
function forgetDecoded() {
|
||||
for (const hint of decodedScopes.keys()) if (!dirty.has(hint) && !scans.has(hint)) decodedScopes.delete(hint);
|
||||
}
|
||||
@@ -92,14 +99,14 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
function dirtyScopes() {
|
||||
return [...dirty].filter(hint => hint !== "*" || !scopeFailures.has(hint))
|
||||
.filter(hint => !scans.has(hint) || scopeFailures.has(hint))
|
||||
.filter(hint => (reconstructAfter.get(hint) ?? 0) <= Date.now() || scopeFailures.has(hint))
|
||||
.filter(hint => gate(hint) <= Date.now() || scopeFailures.has(hint))
|
||||
.map(hint => { try { return decode(hint); } catch { return undefined; } });
|
||||
}
|
||||
function next() {
|
||||
if (!canRun()) return dirty.size > 0 || entries.size > 0 ? Date.now() + 30_000 : null;
|
||||
let nextAt: number | null = null;
|
||||
const take = (at: number) => { nextAt = nextAt === null ? at : Math.min(nextAt, at); };
|
||||
for (const hint of dirty) take(Math.max(earliest, reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0));
|
||||
for (const hint of dirty) take(Math.max(earliest, gate(hint), scopeFailures.get(hint)?.retryAt ?? 0));
|
||||
const unavailable = matcher(dirtyScopes());
|
||||
for (const entry of entries.values()) if (!unavailable(entry))
|
||||
take(Math.max(earliest, entry.dueAt, failures.get(entry.id)?.retryAt ?? 0));
|
||||
@@ -119,7 +126,7 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
for (const [hint, until] of reconstructAfter) {
|
||||
if (until <= Date.now()) reconstructAfter.delete(hint);
|
||||
}
|
||||
const ready = [...dirty].filter(hint => Math.max(reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now());
|
||||
const ready = [...dirty].filter(hint => Math.max(gate(hint), scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now());
|
||||
if (ready.length === 0 && ![...entries.values()].some(entry => Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now())) return;
|
||||
earliest = Date.now() + (options.minimumPassIntervalMs ?? 1000);
|
||||
try { await options.beforePass?.(); } catch (error) {
|
||||
@@ -325,7 +332,7 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
},
|
||||
// Retain scopes/deadlines for resume without retaining a settled write's
|
||||
// idle hold during drain. The coordinator still fences unresolved XIDs.
|
||||
hasPending: async () => canRun() && dirty.size > 0,
|
||||
hasPending: async () => canRun() && [...dirty].some(holdsIdle),
|
||||
nextRunAt: next,
|
||||
nextRunAtOnError: next,
|
||||
});
|
||||
@@ -337,7 +344,7 @@ export function registerScopedDeadlineWork<Entry extends ScheduledEntry, Scope>(
|
||||
return { ...worker,
|
||||
reconciliationSnapshot: (): DeadlineReconciliationSnapshot => ({
|
||||
revision: `${ownerRevision}:${revision}`,
|
||||
reconciled: !stopped && !running && !lastRunFailed && dirty.size === 0 && scans.size === 0
|
||||
reconciled: !stopped && !running && !lastRunFailed && ![...dirty].some(holdsIdle) && scans.size === 0
|
||||
&& failures.size === 0 && scopeFailures.size === 0,
|
||||
}),
|
||||
wake() { revision++; worker.wake(); },
|
||||
|
||||
@@ -6,7 +6,7 @@ import { and, asc, desc, eq, gt, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { chatConversations, chatEndpoints, heartbeatRuns, issueComments, issues, type Db } from "@paperclipai/db";
|
||||
import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
|
||||
|
||||
export type ExternalConversationSettlementScope = { companyId: string; issueId?: string };
|
||||
export type ExternalConversationSettlementScope = { companyId: string; issueId?: string; runId?: string };
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { and, eq, inArray, isNotNull, isNull, ne, or, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, issueComments, issues, type Db } from "@paperclipai/db";
|
||||
import { agentWakeupRequests, budgetPolicies, issueComments, issues, type Db } from "@paperclipai/db";
|
||||
import { notifyDeferredExecutionEligibility, notifyDeferredExecutionWork, notifyDeferredWakesForIssue } from "./deferred-execution-notifications.js";
|
||||
import { withQueuedCommentIdsInWakePayload } from "./issue-queued-comment-queue.js";
|
||||
import { deliveredQueuedCommentIds } from "./queued-comment-delivery.js";
|
||||
@@ -65,12 +65,17 @@ const DEFERRED_RETRY_BACKOFF = sql`interval '30 seconds'`;
|
||||
* issue by id. All notifications publish on commit.
|
||||
*/
|
||||
export async function promoteDeferredWorkIntents(tx: object, input: {
|
||||
companyId: string; trigger: WorkIntentHoldTrigger; issueId?: string | null;
|
||||
companyId: string; trigger: WorkIntentHoldTrigger; issueId?: string | null; runId?: string | null;
|
||||
}) {
|
||||
const hardStop = input.trigger === "accounting_settled" && input.runId ? (await (tx as Pick<Db, "select">)
|
||||
.select({ id: budgetPolicies.id }).from(budgetPolicies).where(and(eq(budgetPolicies.companyId, input.companyId),
|
||||
eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true))).limit(1)).length > 0 : true;
|
||||
const scope = input.trigger === "lock_released" && input.issueId ? { companyId: input.companyId, issueId: input.issueId }
|
||||
: input.trigger === "accounting_settled" && input.runId && !hardStop ? { companyId: input.companyId, runId: input.runId } : null;
|
||||
// Register the work signals before the write they announce.
|
||||
await notifyRunDispatch(tx, { companyId: input.companyId });
|
||||
await notifyRunDispatch(tx, scope ?? { companyId: input.companyId });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: input.companyId });
|
||||
await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
if (!scope) await notifyIssueRecovery(tx, { companyId: input.companyId });
|
||||
if (input.issueId) await notifyDeferredWakesForIssue(tx, { companyId: input.companyId, issueId: input.issueId });
|
||||
const db = tx as Pick<Db, "update">;
|
||||
await db.update(agentWakeupRequests).set({
|
||||
|
||||
Reference in new issue
Block a user