mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - Early hand-off at shutdown (`heartbeat/recovery.ts`, `server/src/index.ts`): in deploy-overlap mode, the outgoing server releases the remote controller lease of its live runs at SIGTERM (`recoveryState` `awaiting_runner_reattach`). A run another server already took over is left alone. The incoming server no longer waits for the lease to lapse. - Reattach fast path: - `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`: a reattach starts without waiting for the existing log to be re-verified. - `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that meets a busy agent directory waits in process (at most 30 s) instead of being deferred to a full re-preparation. An ordinary start is still deferred at once, with no added wait. - Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is a worker on the shared delivery coordinator, queue `reattach-phase`. - Trigger: one startup listing of a previous server's live runs; then the hand-over notification (`notifyReattachHandOver`, sent in the hand-over transaction); the earliest lease deadline it watches (a killed owner); and a 2 s re-check only while handed-over runs are still pending, which ends with the phase (cap 180 s). The bounded re-check exists because a run can also end on another path that sends no notification to this queue. - Source of truth: `native_run_finalizations` and `heartbeat_runs` rows. - Quiet condition: nothing pending and no watched lease, so no deadline and no SQL. It stops with the coordinator at shutdown. - Reached hook for every driver (`native-session-executor.ts`): a reattached run counts as reached when the native session runtime hands the executor its recovered or opened session (`onSession`). Every driver reaches that point only after the Runner answered this server's attach and recovery commands, including an adopted `RunnerdExternalDriver` Runner, which records no new process identity. The identity hook stays as an earlier signal. - Claim wake: when the reattach-phase worker finds runs that just became claimable (hand-over notification or a killed owner's lease deadline), it starts this server's restart-recovery claim for them at once. A validation cell showed the claim otherwise waited about 21.5 s after the hand-over. - `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase` return now sits before every action path of `act()`, except starting an overdue saved-work recovery (it resumes an existing run). - The in-transaction admission re-check of a queued claim (`confirmRunAdmissionStillOpen`) also honours the reattach phase. A phase that begins during a claim returns the run to the queue through the existing rollback. - A reattach whose log re-verification runs in the background awaits it before the log is finalized (`settleDeferredLogRecovery`, both finalization sites in `run-completion.ts`). - Reattach phase (`heartbeat/reattach-phase.ts`, `heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while runs that a previous controller handed over (lease released or observed expired, or claimed by this server's restart recovery) are claimable but not yet reached. A run is reached when its process identity is recorded. Authentication is logged as a mark. The cap (180 s, `PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs became claimable. Later batches join or re-arm the phase. Runs reached after 150 s are counted and logged. - Gates on the phase: - `claimQueuedRun`, start-next, resume-queued and dispatch in `heartbeat/queue.ts`, `run-lifecycle.ts` and `native-runner-execution-driver.ts` leave queued runs queued. - `execution-recovery-resolution.ts`: one line, the first line of `deliverReconciledExecutions`. - `recovery/assigned-work-sweep.ts`: one line, outcome `deferred_reattach_phase`. - `heartbeat/recovery.ts`: a one-line claim hook (`noteReattachRunsClaimed`) after restart recovery claims runs. - The owner of those three files agreed to these one-line gates. - `tests/restart-chaos/daytona-image.ts`: `PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a controller-only branch against an existing Daytona image. It accepts a content-ID mismatch only for the exact content ID of the named image, and it prints a TEST OVERRIDE warning. This is harness-only. ## Verification Red on master, green on this branch: - `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six two-server scenarios were red on the earlier phase timing: - `does not hold queued starts while the previous owner is alive and not handing over` - `closes the gate when the previous owner hands its runs over, and counts reach latency from the hand-over` - `runs the cap from the hand-over, so a wedged run ends the phase without the overlap eating it` - `begins when a killed owner's lease is observed expired, without any hand-over` - `re-arms for a later batch, including runs the previous owner started after discovery` - `closes the gate at once when this server claims a run between two lease reads, dated from the hand-over` - plus `claimQueuedRun leaves a queued run queued and unstarted, in the admission gate's shape`, `is the same gate the assigned-work sweep and reconciled continuation delivery read`, `ends at the cap, counted from when the runs became claimable, so one wedged Runner cannot hold run starts` - `heartbeat-native-early-hand-off.test.ts`: `releases the remote controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56 ms), `leaves a run another server already took over alone`, `holds the incoming server's queued starts only from the hand-over, not while the outgoing server still drives its runs` - `heartbeat/run-output.test.ts`: `lets a reattach start before its existing log is re-verified` - `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at once, with no wait added to the run-start path`, `lets a reattach wait its turn in process instead of a deferred full re-preparation`, `still defers a reattach once its bounded wait runs out` - `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only for the exact image content ID named by the test override` - Review fixes (red before the fix, green after): `reattach-phase.test.ts` `starts the claim of handed-over runs within a second of the hand-over notification`; `assigned-work-invariant.test.ts` `takes no action while handed-over live runs are being reached, and acts once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never lets the outgoing server claim back a run it handed off, while the incoming server may take it over`; `reattach-phase.test.ts` `retries a failed first listing instead of going quiet, and still finds the previous owner's runs`, `keeps a hand-over notification whose listing failed until a listing succeeds`, `moves the cap earlier when a claimed run turns out to have been handed over before its claim`; `queue.test.ts` `returns the claim to the queue when a reattach phase begins while the claim is being checked`; `run-completion.test.ts` `finalizes the run log in completeRun|failRun only after a reattach's deferred log recovery settles`. `native-session-executor.test.ts` `ends the reattach phase once a reattached %s session is driven by this server again, without the cap or manual settling` over codex_app_server, opencode_server, acpx_runtime, muse_external, grok_bot_mcp and openai_dot_mcp (6 of 6 red without the hook; the test drives the executor with a mocked session runtime, not each real driver). Lifecycle tests for the new worker: `does no database work and keeps no deadline on an instance with nothing to reattach`, `waits for the hand-over notification or the lease deadline instead of polling while the previous owner drives its runs`, `begins at a killed owner's lease deadline, without any hand-over or notification`, `stops with the delivery coordinator at shutdown and ignores a later hand-over`. Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing on this branch: `reattach-phase.test.ts`, `heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53 tests together), `agent-directory-busy.test.ts`, `heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30 to 96 on this shared machine, the full `queue.test.ts` and `recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres setup; CI supplies them. Daytona results that exist (local servers with Daytona sandboxes, release-swap cells). They ran on a combined branch with this PR, #21462, #21442 and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (masterd1d71a7755): SIGTERM to PRP authentication 21.5 to 38 s at swap 1. - G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7 s at swap 1. All runs that crossed swap 2 were adopted on the same Runner process. The staging chain (`provider recovery timed out after 900000ms`, then `resumed_not_suspended`) did not recur. - Caveat: G5c ran before the phase timing fix in this PR. In G5c the phase opened at process start, its 180 s cap ran out during the overlap, and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease claims. The timing fix (the phase begins when the runs become claimable) has not run on Daytona. - G6c (12 live runs and 20 queued, two graceful swaps, no injection, combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11 survivors at both. - Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after SIGTERM. The 30 s target was met. - Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at 19 s, so the outgoing server drained gracefully. The incoming server did not claim on the hand-over; it claimed only when the outgoing server exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity 47 to 57 s after SIGTERM. The target was missed. Why the claims waited for the exit is not explained. - The host load was 49 to 70 from other work. 7 of 32 tasks timed out in the queue or budget after the swaps; we suspect the overload. ## Risks - Product trade-off: while handed-over live runs are being reached, queued starts wait, up to the 180 s cap. On an instance with no handed-over runs, nothing waits. - Not proven: two-server behavior with the phase timing fix and the notification-driven watch (no cell has run them); warm standby becoming active; runs that the outgoing server starts after discovery. - Early hand-off at SIGTERM orphans a server-side exec tool that is in flight. With #21442 the next controller settles it as interrupted with an unknown outcome; it is never run again. - Swap 2 in G6c: the incoming server claimed only at the outgoing server's exit. That is unexplained. - Cross-session ownership: the gates in `execution-recovery-resolution.ts` and `recovery/assigned-work-sweep.ts`, and the claim hook in `heartbeat/recovery.ts`, are in files that the restart-recovery work owns. They are one-line gates agreed with that owner. That owner said the gate did not need to be inside the claim transaction; that was before the phase could begin during a claim. After review, the in-transaction re-check in `heartbeat/queue.ts` also reads the phase, through the existing rollback. #21427 (open) also edits `execution-recovery-resolution.ts`, `heartbeat/queue.ts` and `recovery/assigned-work-sweep.ts`; whichever merges second needs a rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned by the restart-recovery work): a 3-line guard in `claimNativeRestartRecoveries` returns a run this process handed off at shutdown as `blocked` / `handed_off_to_next_controller` before takeover is evaluated, so the outgoing server never claims back a run it handed off. #21478 (merged) added a sweep path that starts an overdue saved-work recovery; it is deliberately not gated by the reattach phase, because it resumes an existing run (reattach-like work), which the phase makes room for rather than competes with. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
849 lines
59 KiB
TypeScript
849 lines
59 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql as sqlTag } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { enableReattachPhase, joinReattachPhase, resetReattachPhaseForTests, settleReattachRun } from "../services/heartbeat/run-admission.js";
|
|
import {
|
|
activityLog, agents, agentTaskSessions, agentWakeupRequests, approvals, companies, createDb, heartbeatRuns, issueApprovals,
|
|
issueRecoveryActions, issueRelations, issues, issueThreadInteractions, nativeRunFinalizations,
|
|
} from "@paperclipai/db";
|
|
import { recoveryService } from "../services/recovery/service.js";
|
|
import { heartbeatService } from "../services/heartbeat.js";
|
|
import { assignedWorkSweep, ASSIGNED_WORK_CONTINUED_ACTION, ASSIGNED_WORK_REPAIRS_PER_MINUTE } from "../services/recovery/assigned-work-sweep.js";
|
|
import { CONTINUATION_CAP } from "../services/recovery/assigned-work-invariant.js";
|
|
import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
|
|
|
|
const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
|
|
if (!recoveryPostgresSupport.supported) console.warn(`Skipping assigned-work invariant tests: ${recoveryPostgresSupport.reason}`);
|
|
|
|
const HOUR = 60 * 60_000;
|
|
|
|
/**
|
|
* The assigned-work invariant on real rows. Each stranded shape below was
|
|
* measured on staging on 2026-11-12; the existing stranded sweep leaves each
|
|
* of them with nothing pending, and the invariant sweep gives each exactly one
|
|
* ordinary continuation wake.
|
|
*/
|
|
describePostgres("assigned-work invariant sweep", () => {
|
|
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startRecoveryPostgres("assigned-work-invariant-");
|
|
db = createDb(temporary.connectionString);
|
|
}, 240_000);
|
|
|
|
afterAll(async () => {
|
|
await db?.$client.end({ timeout: 0 });
|
|
await temporary?.cleanup();
|
|
});
|
|
|
|
/** A wake queue stand-in that records each wake as a durable pending row. */
|
|
function wakeQueue() {
|
|
const calls: Array<{ agentId: string; issueId: string; source: unknown }> = [];
|
|
const enqueueWakeup = vi.fn(async (agentId: string, opts?: { payload?: Record<string, unknown> | null; contextSnapshot?: Record<string, unknown>; reason?: string | null }) => {
|
|
const issueId = String(opts?.payload?.issueId);
|
|
const [issue] = await db.select({ companyId: issues.companyId }).from(issues).where(eq(issues.id, issueId));
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: issue.companyId, agentId, source: "automation",
|
|
triggerDetail: "system", reason: opts?.reason ?? null, payload: opts?.payload ?? null, status: "queued" }).returning();
|
|
calls.push({ agentId, issueId, source: opts?.contextSnapshot?.source });
|
|
return { id: wake.id } as never;
|
|
});
|
|
return { calls, enqueueWakeup };
|
|
}
|
|
|
|
async function seedCompany() {
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Invariant fixture", issuePrefix: `I${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Builder", adapterType: "codex_local", status: "idle", adapterConfig: {} });
|
|
let number = 0;
|
|
async function issue(values: Partial<typeof issues.$inferInsert> = {}) {
|
|
const id = randomUUID();
|
|
number += 1;
|
|
await db.insert(issues).values({ id, companyId, title: `Task ${number}`, status: "in_progress", assigneeAgentId: agentId,
|
|
issueNumber: number, identifier: `I${companyId.slice(0, 7)}-${number}`, updatedAt: new Date(Date.now() - 3 * HOUR),
|
|
createdAt: new Date(Date.now() - 5 * HOUR), ...values });
|
|
return id;
|
|
}
|
|
async function run(issueId: string, values: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
|
const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "interrupted",
|
|
errorCode: "server_shutdown_interrupted", startedAt: new Date(Date.now() - 4 * HOUR), finishedAt: new Date(Date.now() - 3 * HOUR),
|
|
livenessState: "needs_followup", contextSnapshot: { issueId, taskId: issueId }, createdAt: new Date(Date.now() - 4 * HOUR),
|
|
...values }).returning();
|
|
return row;
|
|
}
|
|
return { companyId, agentId, issue, run };
|
|
}
|
|
|
|
it("a deploy-interrupted in_progress task with nothing pending gets exactly one continuation; the stranded sweep leaves it", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
// The staging run: a person's Retry, stopped by a deploy's shutdown during
|
|
// setup, certified as no provider work, its hold already resolved.
|
|
await f.run(issueId, { resultJson: { stopReason: "interrupted", workspaceRestoreFailure: "restore_failed",
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
contextSnapshot: { issueId, taskId: issueId, explicitUserContinuation: { previousRunId: randomUUID() } } });
|
|
const queue = wakeQueue();
|
|
// Production wiring: the bounded retry scheduler declines (not_scheduled).
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId, skipRelatedRecovery: true });
|
|
expect(queue.calls).toHaveLength(0);
|
|
const [stillStranded] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(stillStranded.status).toBe("in_progress");
|
|
|
|
const check = await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId });
|
|
// What staging shows for this shape: a Retry button that only the system can act on.
|
|
expect(check).toMatchObject({ outcome: "repaired", verdict: { kind: "violation", reason: "retry_offered" } });
|
|
expect(queue.calls).toEqual([{ agentId: f.agentId, issueId, source: "issue.assigned_work_invariant" }]);
|
|
const [activity] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
|
|
expect(activity.details).toMatchObject({ message: expect.stringContaining("Continued automatically") });
|
|
expect(String(activity.details?.message)).not.toMatch(/_|[0-9a-f]{8}-/);
|
|
|
|
// The next sweep sees the pending wake and leaves the task alone.
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent", intent: "queued_wake" } });
|
|
expect(queue.calls).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("a wake deferred behind a run that ended long ago is not a pending path; a fresh one still is", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId, { status: "failed", errorCode: "native_restart_recovery_blocked" });
|
|
const deferred = (createdAt: Date) => db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", triggerDetail: "system", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "local-board", payload: { issueId }, createdAt, updatedAt: new Date() });
|
|
await deferred(new Date(Date.now() - 30 * 60_000));
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).toContain(issueId);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { kind: "violation", reason: "no_pending_path" } });
|
|
|
|
const freshIssue = await f.issue();
|
|
await f.run(freshIssue, { status: "failed", errorCode: "native_restart_recovery_blocked" });
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
payload: { issueId: freshIssue }, createdAt: new Date(Date.now() - 60_000) });
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).not.toContain(freshIssue);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: freshIssue }))
|
|
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent", intent: "queued_wake" } });
|
|
}, 60_000);
|
|
|
|
it("a wake claimed long ago that never became a run is not a pending path", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId, { status: "succeeded", errorCode: null, livenessState: null });
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_children_completed", status: "claimed", payload: { issueId },
|
|
claimedAt: new Date(Date.now() - 15 * 60_000), createdAt: new Date(Date.now() - 40 * 60_000) });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).toContain(issueId);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ verdict: { kind: "violation", reason: "no_pending_path" } });
|
|
}, 60_000);
|
|
|
|
it("an in_progress task whose native run failed needing reconciliation (silently skipped today) gets one continuation", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation",
|
|
livenessState: "failed" });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId, skipRelatedRecovery: true });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
|
|
expect(queue.calls).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("a task blocked after a legacy run's retries ran out, with no hold left, is restored and continued once", async () => {
|
|
// The shape the fresh-continuation fallback does not cover: retries spent
|
|
// on a restart-caused failure (controller lease lost), block written by
|
|
// the stranded sweep, and the hold already resolved.
|
|
const f = await seedCompany();
|
|
const blockedAt = new Date(Date.now() - 2 * HOUR);
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
const run = await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "stranded_assigned_issue",
|
|
status: "resolved", outcome: "blocked", ownerType: "board", cause: "stranded_assigned_issue",
|
|
fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
|
|
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "controller_lease_lost" },
|
|
nextAction: "Inspect the retry history.", attemptCount: 3, resolvedAt: blockedAt });
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
|
|
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
|
|
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(issueId);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "recovery_block" } });
|
|
expect(queue.calls).toHaveLength(1);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("in_progress");
|
|
}, 60_000);
|
|
|
|
it("a recovery block whose task already has a queued run is lifted without a second run", async () => {
|
|
const f = await seedCompany();
|
|
const blockedAt = new Date(Date.now() - 20 * 60_000);
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
await f.run(issueId, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation",
|
|
livenessState: "failed", finishedAt: new Date(blockedAt.getTime() - 5_000) });
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "execution-recovery", action: "issue.updated",
|
|
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
|
|
reason: "native_continuation_requires_reconciliation" }, createdAt: blockedAt });
|
|
const successor = await f.run(issueId, { status: "queued", errorCode: null, livenessState: null, startedAt: null, finishedAt: null,
|
|
createdAt: new Date(blockedAt.getTime() + 19_000) });
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_continuation_needed", status: "queued", runId: successor.id,
|
|
payload: { issueId } }).returning();
|
|
await db.update(heartbeatRuns).set({ wakeupRequestId: wake.id }).where(eq(heartbeatRuns.id, successor.id));
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(issueId);
|
|
expect(await recovery.assignedWork.report(f.companyId)).toMatchObject({ violations: { blocked_with_live_run: 1 } });
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "blocked_with_live_run", runId: successor.id } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("todo");
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "ok", verdict: { clause: "live_run" } });
|
|
}, 60_000);
|
|
|
|
it("continues a recent board escalation of a temporary setup failure itself, and leaves old or lasting ones", async () => {
|
|
const f = await seedCompany();
|
|
async function escalated(input: { endedAgoMs: number; cause: string }) {
|
|
const blockedAt = new Date(Date.now() - input.endedAgoMs + 1_000);
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
const run = await f.run(issueId, { status: "failed", errorCode: "reusable_sandbox_resume_failed", livenessState: "failed",
|
|
error: `Failed to acquire lease: Reusable sandbox lease "x" could not be resumed (cause=${input.cause}); the lease was preserved and no replacement was created.`,
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
startedAt: new Date(Date.now() - input.endedAgoMs - 60_000), finishedAt: new Date(Date.now() - input.endedAgoMs),
|
|
createdAt: new Date(Date.now() - input.endedAgoMs - 60_000) });
|
|
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId,
|
|
kind: "stranded_assigned_issue", status: "active", ownerType: "board", cause: "stranded_assigned_issue",
|
|
fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
|
|
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "reusable_sandbox_resume_failed",
|
|
routingPolicy: "board_escalation_no_takeover_v1" },
|
|
nextAction: "Board operator: inspect the evidence, repair the runtime if appropriate.", attemptCount: 1 }).returning();
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
|
|
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
|
|
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
|
|
return { issueId, hold };
|
|
}
|
|
const recent = await escalated({ endedAgoMs: 2 * HOUR, cause: "transport_timeout" });
|
|
const old = await escalated({ endedAgoMs: 30 * HOUR, cause: "transport_timeout" });
|
|
const lasting = await escalated({ endedAgoMs: 2 * HOUR, cause: "provider_access_denied" });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const suspects = (await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id);
|
|
expect(suspects).toContain(recent.issueId);
|
|
expect(suspects).not.toContain(old.issueId);
|
|
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: recent.issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "recovery_block" } });
|
|
expect(queue.calls).toEqual([{ agentId: f.agentId, issueId: recent.issueId, source: "issue.assigned_work_invariant" }]);
|
|
expect((await db.select().from(issues).where(eq(issues.id, recent.issueId)))[0].status).toBe("in_progress");
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, recent.hold.id)))[0])
|
|
.toMatchObject({ status: "resolved", outcome: "restored", ownerType: "system", resolutionNote: "stranded_transient_continued" });
|
|
|
|
for (const left of [old, lasting]) {
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: left.issueId }))
|
|
.toMatchObject({ outcome: "left_to_hold_owner" });
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, left.hold.id)))[0].status).toBe("active");
|
|
}
|
|
expect(queue.calls).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("leaves a task blocked under an active recovery hold to the hold's owner and reports it", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 2 * HOUR) });
|
|
const run = await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed",
|
|
startedAt: new Date(Date.now() - 31 * HOUR), finishedAt: new Date(Date.now() - 30 * HOUR), createdAt: new Date(Date.now() - 31 * HOUR) });
|
|
// Old-shaped row: a board-owned stranded action written by an older release.
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "stranded_assigned_issue",
|
|
status: "active", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
|
|
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "process_lost" },
|
|
nextAction: "Board operator: inspect the retry history, then explicitly retry the original owner, reassign, or intentionally resolve the task.",
|
|
attemptCount: 1 });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "left_to_hold_owner", verdict: { kind: "violation", reason: "recovery_block" } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(issueId);
|
|
expect(await recovery.assignedWork.report(f.companyId)).toMatchObject({ violationCount: 1, heldByRecoveryOwner: 1 });
|
|
}, 60_000);
|
|
|
|
it("Retry-started tasks the scheduler refused to revalidate get one continuation, even with a question asked before the Retry", async () => {
|
|
// Nine staging tasks: a person's Retry, stopped by a deploy during setup,
|
|
// certified as no provider work; the scheduler then refused with
|
|
// continuation_user_authorization_missing on every sweep. One of them
|
|
// also had a question the agent asked before the person pressed Retry.
|
|
const f = await seedCompany();
|
|
const plain = await f.issue();
|
|
const withQuestion = await f.issue();
|
|
for (const issueId of [plain, withQuestion]) {
|
|
await f.run(issueId, { resultJson: { stopReason: "interrupted", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
contextSnapshot: { issueId, taskId: issueId, explicitUserContinuation: { previousRunId: randomUUID() } },
|
|
createdAt: new Date(Date.now() - 4 * HOUR) });
|
|
}
|
|
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: withQuestion, kind: "ask_user_questions",
|
|
status: "pending", continuationPolicy: "wake_assignee", payload: {}, createdAt: new Date(Date.now() - 12 * HOUR) } as never);
|
|
const queue = wakeQueue();
|
|
// On staging the refusal also left the sweep's escalation without effect
|
|
// (its stale-snapshot guard): the tasks stayed in_progress with nothing
|
|
// pending. Model that by changing the task under the sweep's snapshot.
|
|
const scheduleRecoveryRetry = vi.fn(async (runId: string, outcome?: { continuationRefused?: boolean }) => {
|
|
if (outcome) outcome.continuationRefused = true;
|
|
const [run] = await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
await db.update(issues).set({ updatedAt: new Date(Date.now() - 2 * HOUR) }).where(eq(issues.id, String(run.context?.issueId)));
|
|
return null;
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry });
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, skipRelatedRecovery: true });
|
|
expect(queue.calls).toHaveLength(0);
|
|
|
|
for (const issueId of [plain, withQuestion]) {
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("in_progress");
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "retry_offered" } });
|
|
}
|
|
expect(queue.calls.map(call => call.issueId).sort()).toEqual([plain, withQuestion].sort());
|
|
}, 60_000);
|
|
|
|
it("an unknown block reason is repaired, not treated as a reason to wait", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
|
|
await f.run(issueId, { status: "failed", errorCode: "something_new", livenessState: "failed" });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "unknown_block" } });
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("todo");
|
|
expect(queue.calls).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("two servers sweeping the same violation at once create one wake", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
const queue = wakeQueue();
|
|
const serverA = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const serverB = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const results = await Promise.all([serverA.checkIssue({ companyId: f.companyId, issueId }), serverB.checkIssue({ companyId: f.companyId, issueId })]);
|
|
expect(queue.calls).toHaveLength(1);
|
|
expect(results.map(result => result?.outcome).sort()).toEqual(["repaired", expect.stringMatching(/changed|deferred_rate_limit/)].sort());
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId), eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))))
|
|
.toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("leaves a run owned by another live server alone, and settles one whose owner is proven gone", async () => {
|
|
const f = await seedCompany();
|
|
const otherBoot = randomUUID();
|
|
const controllerProcess = { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() };
|
|
// Deploy overlap: the other server holds the lease.
|
|
const leased = await f.issue();
|
|
await f.run(leased, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
|
|
controllerLeaseExpiresAt: new Date(Date.now() + 60_000) });
|
|
// Lease lapsed, but the owner proof says its server is still alive.
|
|
const lapsedAlive = await f.issue();
|
|
await f.run(lapsedAlive, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
|
|
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
|
|
const queue = wakeQueue();
|
|
const alive = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup,
|
|
proveOwner: async () => ({ verdict: "alive", reason: "controller_lease_held" }) });
|
|
expect(await alive.checkIssue({ companyId: f.companyId, issueId: leased })).toMatchObject({ verdict: { clause: "live_run" } });
|
|
expect(await alive.checkIssue({ companyId: f.companyId, issueId: lapsedAlive })).toMatchObject({ verdict: { clause: "live_run" } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
|
|
const settle = vi.fn(async () => undefined);
|
|
const gone = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
|
|
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }) });
|
|
expect(await gone.checkIssue({ companyId: f.companyId, issueId: leased })).toMatchObject({ verdict: { clause: "live_run" } });
|
|
const goneCheck = await gone.checkIssue({ companyId: f.companyId, issueId: lapsedAlive });
|
|
expect(goneCheck)
|
|
.toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone" } });
|
|
expect(settle).toHaveBeenCalledTimes(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("takes no action while handed-over live runs are being reached, and acts once the phase ends", async () => {
|
|
const f = await seedCompany();
|
|
const otherBoot = randomUUID();
|
|
const controllerProcess = { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() };
|
|
const lapsed = await f.issue();
|
|
await f.run(lapsed, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
|
|
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
|
|
const queue = wakeQueue();
|
|
const settle = vi.fn(async () => undefined);
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
|
|
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }) });
|
|
try {
|
|
enableReattachPhase(180_000);
|
|
joinReattachPhase([{ runId: "handed-over-live-run", reason: "handed_over", claimableAt: Date.now() }]);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: lapsed }))
|
|
.toMatchObject({ outcome: "deferred_reattach_phase", verdict: { reason: "run_owner_gone" } });
|
|
expect(settle).not.toHaveBeenCalled();
|
|
expect(queue.calls).toHaveLength(0);
|
|
settleReattachRun("handed-over-live-run", "reached");
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: lapsed }))
|
|
.toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone" } });
|
|
expect(settle).toHaveBeenCalledTimes(1);
|
|
} finally {
|
|
resetReattachPhaseForTests();
|
|
}
|
|
}, 60_000);
|
|
|
|
it("leaves tasks waiting on an approval, a question, a dependency or a declared unblock action", async () => {
|
|
const f = await seedCompany();
|
|
const approvalTask = await f.issue();
|
|
const [approval] = await db.insert(approvals).values({ companyId: f.companyId, type: "plan", status: "pending", payload: {} } as never).returning();
|
|
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: approvalTask, approvalId: approval.id } as never);
|
|
const questionTask = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
|
|
await f.run(questionTask, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation" });
|
|
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: questionTask, kind: "request_confirmation",
|
|
status: "pending", continuationPolicy: "wake_assignee", payload: {} } as never);
|
|
const dependent = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
|
|
const blocker = await f.issue({ status: "todo", assigneeAgentId: null });
|
|
await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blocker, relatedIssueId: dependent, type: "blocks" } as never);
|
|
const declared = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR),
|
|
unblockDescriptor: { owner: "board", action: "Share the deploy key for the staging project." } });
|
|
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: approvalTask })).toMatchObject({ verdict: { reason: "approval" } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: questionTask })).toMatchObject({ verdict: { reason: "question" } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: dependent })).toMatchObject({ verdict: { reason: "dependency" } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: declared })).toMatchObject({ verdict: { reason: "assignee_declared_block" } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
// The indexed pre-filter leaves all four out of the sweep entirely.
|
|
const suspects = await sweep.listSuspects({ companyId: f.companyId }, null, 100);
|
|
expect(suspects.map(row => row.id)).not.toEqual(expect.arrayContaining([approvalTask]));
|
|
for (const id of [approvalTask, questionTask, dependent, declared]) expect(suspects.map(row => row.id)).not.toContain(id);
|
|
}, 60_000);
|
|
|
|
it("a refused or later-declined continuation stays a violation, says so, and is tried again under the cap", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
let admit = false;
|
|
const enqueueWakeup = vi.fn(async (agentId: string, opts?: { payload?: Record<string, unknown> | null; reason?: string | null }) => {
|
|
if (!admit) return null;
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId, source: "automation",
|
|
triggerDetail: "system", reason: opts?.reason ?? null, payload: opts?.payload ?? null, status: "queued" }).returning();
|
|
return { id: wake.id } as never;
|
|
});
|
|
let clock = new Date();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup, now: () => clock });
|
|
const deadlines: number[] = [];
|
|
// Admission refuses: the entry says nothing started, and a re-check is due.
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) }))
|
|
.toMatchObject({ outcome: "enqueue_refused" });
|
|
const activity = async () => db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
|
|
expect((await activity())[0].details).toMatchObject({ outcome: "not_started", message: expect.stringContaining("could not start") });
|
|
expect(deadlines).toHaveLength(1);
|
|
// Too soon: spaced, not repaired and not reported as fine.
|
|
clock = new Date(clock.getTime() + 60_000);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "deferred_rate_limit", verdict: { kind: "violation" } });
|
|
// After the spacing it is admitted; the queue later declines the claim
|
|
// (for example behind the old run's pending cleanup) and the wake is skipped.
|
|
admit = true;
|
|
clock = new Date(clock.getTime() + 2 * 60_000);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
|
|
expect((await activity()).map(row => row.details?.outcome).sort()).toEqual(["not_started", "queued"]);
|
|
await db.update(agentWakeupRequests).set({ status: "skipped" }).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
clock = new Date(clock.getTime() + 5 * 60_000);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
|
|
expect(enqueueWakeup).toHaveBeenCalledTimes(3);
|
|
}, 60_000);
|
|
|
|
it("with the real wake queue, a hold reached by the sweep's wake becomes one pending fresh continuation", async () => {
|
|
// The task's last run left a legacy reconciliation hold. The sweep's
|
|
// ordinary continuation wake reaches the fresh-continuation fallback,
|
|
// which settles the hold and records one pending continuation (owned by
|
|
// the execution-continuation worker). The next sweep adds no attempt.
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
const run = await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed" });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
status: "active", ownerType: "board", cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${run.id}`,
|
|
evidence: { runId: run.id, originalFailureCode: "process_lost" }, nextAction: "Reconcile the stopped run.", attemptCount: 1 });
|
|
const heartbeat = heartbeatService(db);
|
|
try {
|
|
const first = await heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId });
|
|
expect(first).toMatchObject({ outcome: "enqueue_refused", verdict: { kind: "violation" } });
|
|
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
sqlTag`${heartbeatRuns.status} in ('queued', 'running', 'scheduled_retry')`));
|
|
expect(queued).toHaveLength(0);
|
|
const pending = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
|
sqlTag`${agentWakeupRequests.status} in ('queued', 'deferred_issue_execution', 'claimed')`));
|
|
expect(pending).toHaveLength(0);
|
|
const [entry] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
|
|
expect(entry.details).toMatchObject({ outcome: "not_started" });
|
|
const [record] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
sqlTag`${issueRecoveryActions.evidence}->'freshContinuation' is not null`));
|
|
expect(record!.evidence).toMatchObject({ continuationDelivery: "pending" });
|
|
// A second sweep right away adds no attempt.
|
|
expect(await heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId }))
|
|
.not.toMatchObject({ outcome: "repaired" });
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)))).toHaveLength(1);
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
}
|
|
}, 120_000);
|
|
|
|
it("routine touches on a stranded task (a comment, a read, a label) do not postpone its repair", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, issueId));
|
|
const queue = wakeQueue();
|
|
expect(await assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup }).checkIssue({ companyId: f.companyId, issueId }))
|
|
.toMatchObject({ outcome: "repaired" });
|
|
expect(queue.calls).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("restores a block made through the normal update route to its recorded status, and a review block back to review without a wake", async () => {
|
|
const f = await seedCompany();
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const blockedAt = new Date(Date.now() - 2 * HOUR);
|
|
const cases = [["in_progress", "in_progress", 1], ["in_review", "in_review", 0]] as const;
|
|
for (const [before, expected, wakes] of cases) {
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed" });
|
|
// The shape the normal issue update route writes.
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
|
|
entityType: "issue", entityId: issueId, details: { status: "blocked", _previous: { status: before } }, createdAt: blockedAt });
|
|
const callsBefore = queue.calls.length;
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe(expected);
|
|
expect(queue.calls.length - callsBefore).toBe(wakes);
|
|
}
|
|
}, 60_000);
|
|
|
|
it("puts a lifted block back when admission refuses the continuation", async () => {
|
|
const f = await seedCompany();
|
|
const blockedAt = new Date(Date.now() - 2 * HOUR);
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
|
|
const notifiedAt = new Date(Date.now() - HOUR);
|
|
const descriptor = { owner: { agentId: f.agentId }, action: "Retry the build once the cache is warm." };
|
|
await db.update(issues).set({ unblockDescriptor: descriptor as never, blockedOwnerNotifiedAt: notifiedAt }).where(eq(issues.id, issueId));
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: async () => null });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "enqueue_refused" });
|
|
const [after] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(after.status).toBe("blocked");
|
|
expect(after.blockedTransitionAt?.getTime()).toBe(blockedAt.getTime());
|
|
expect(after.unblockDescriptor).toEqual(descriptor);
|
|
expect(after.blockedOwnerNotifiedAt?.getTime()).toBe(notifiedAt.getTime());
|
|
}, 60_000);
|
|
|
|
it("re-runs the predicate under the row lock: a question asked after the first read stops the repair", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
// The first read sees a violation; the person's question lands just before
|
|
// the claim's transaction takes the row lock.
|
|
const original = db.transaction.bind(db);
|
|
const spy = vi.spyOn(db, "transaction").mockImplementationOnce((async (work: never, config?: never) => {
|
|
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId, kind: "ask_user_questions",
|
|
status: "pending", continuationPolicy: "wake_assignee", payload: {} } as never);
|
|
return original(work, config);
|
|
}) as never);
|
|
try {
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "changed", verdict: { kind: "violation" } });
|
|
} finally { spy.mockRestore(); }
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)))).toHaveLength(0);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("reports only tasks the caller may read; the sweep still evaluates private tasks", async () => {
|
|
const f = await seedCompany();
|
|
const open = await f.issue();
|
|
await f.run(open);
|
|
const hidden = await f.issue();
|
|
await f.run(hidden);
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
// A caller whose read condition excludes the private task.
|
|
const report = await sweep.report(f.companyId, { readCondition: sqlTag<boolean>`${issues.id} <> ${hidden}` });
|
|
expect(report).toMatchObject({ checked: 1, violationCount: 1 });
|
|
expect(report.sample.map(row => row.identifier)).not.toContain((await db.select().from(issues).where(eq(issues.id, hidden)))[0].identifier);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: hidden })).toMatchObject({ outcome: "repaired" });
|
|
}, 60_000);
|
|
|
|
it("completes a claim on a one-connection pool: the locked re-check reads through the transaction", async () => {
|
|
const single = createDb(temporary.connectionString, { maxConnections: 1 });
|
|
try {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 2 * HOUR) });
|
|
await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(single, { enqueueWakeup: queue.enqueueWakeup });
|
|
const result = await Promise.race([sweep.checkIssue({ companyId: f.companyId, issueId }),
|
|
new Promise(resolve => setTimeout(() => resolve("deadlocked"), 20_000))]);
|
|
expect(result).toMatchObject({ outcome: "repaired" });
|
|
expect(queue.calls).toHaveLength(1);
|
|
} finally { await single.$client.end({ timeout: 0 }); }
|
|
}, 60_000);
|
|
|
|
it("says when the report is capped, and that the violation count is then a lower bound", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
const report = await assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup }).report(f.companyId);
|
|
expect(report).toMatchObject({ checked: 1, openTasks: 1, truncated: false, violationCountIsLowerBound: false, violationCount: 1 });
|
|
}, 60_000);
|
|
|
|
it("a running run whose owner is proven gone is settled first, then the task gets its continuation within the bound", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
const otherBoot = randomUUID();
|
|
const run = await f.run(issueId, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot,
|
|
controllerProcess: { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() },
|
|
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
|
|
const queue = wakeQueue();
|
|
// Heartbeat's lost-process settle path ends the run, as it does in production.
|
|
const settle = vi.fn(async ({ runId }: { runId: string }) => {
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
|
});
|
|
let clock = new Date();
|
|
const started = clock.getTime();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
|
|
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }), now: () => clock });
|
|
const deadlines: number[] = [];
|
|
const pass = () => sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) });
|
|
// Pass 1: the owner is proven gone outside any lock; the run goes to the settle path, no claim is tried.
|
|
expect(await pass()).toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone", runId: run.id } });
|
|
expect(settle).toHaveBeenCalledTimes(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
// Pass 2 at its deadline: the run has ended and the task is repaired.
|
|
clock = new Date(deadlines.at(-1)! + 1_000);
|
|
expect(await pass()).toMatchObject({ outcome: "repaired" });
|
|
expect(queue.calls).toHaveLength(1);
|
|
expect(clock.getTime() - started).toBeLessThan(5 * 60_000);
|
|
}, 60_000);
|
|
|
|
it("does not count a native run left running in awaiting evidence with no renewal as live, and reports it with its agent's held slot", async () => {
|
|
// The graceful hand-off repro: the new server's startup recovery parked
|
|
// the run in awaiting evidence; nothing renews or settles it.
|
|
const f = await seedCompany();
|
|
const stuck = await f.issue();
|
|
const stuckRun = await f.run(stuck, { runtimeMode: "native", status: "running", errorCode: null, finishedAt: null, nativeIssueId: stuck });
|
|
const old = new Date(Date.now() - 10 * 60_000);
|
|
await db.insert(nativeRunFinalizations).values({ runId: stuckRun.id, companyId: f.companyId, issueId: stuck, phase: "observed",
|
|
recoveryState: "awaiting_evidence", leaseExpiresAt: old, updatedAt: old });
|
|
// A recent claim is still a live path.
|
|
const fresh = await f.issue();
|
|
const freshRun = await f.run(fresh, { runtimeMode: "native", status: "running", errorCode: null, finishedAt: null, nativeIssueId: fresh });
|
|
await db.insert(nativeRunFinalizations).values({ runId: freshRun.id, companyId: f.companyId, issueId: fresh, phase: "observed",
|
|
recoveryState: "awaiting_evidence", leaseExpiresAt: new Date(Date.now() + 60_000) });
|
|
// Queued work for the same agent that cannot dispatch behind the held slot.
|
|
const waiting = await f.issue({ status: "todo" });
|
|
await f.run(waiting, { status: "queued", errorCode: null, finishedAt: null, startedAt: null });
|
|
const queue = wakeQueue();
|
|
const settle = vi.fn(async () => undefined);
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: stuck }))
|
|
.toMatchObject({ outcome: "left_to_native_recovery", verdict: { kind: "violation", reason: "native_run_stalled", runId: stuckRun.id } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: fresh })).toMatchObject({ verdict: { clause: "live_run" } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(settle).not.toHaveBeenCalled();
|
|
const report = await sweep.report(f.companyId);
|
|
expect(report.violations).toMatchObject({ native_run_stalled: 1 });
|
|
expect(report.agentsWithStalledSlots).toEqual([{ agentName: "Builder", stalledRuns: 1, queuedRuns: 1 }]);
|
|
// A caller who cannot read those tasks sees none of their runs counted.
|
|
const hiddenIds = [stuck, waiting];
|
|
const filtered = await sweep.report(f.companyId, { readCondition: sqlTag<boolean>`${issues.id} not in (${sqlTag.join(hiddenIds.map(id => sqlTag`${id}::uuid`), sqlTag`, `)})` });
|
|
expect(filtered.violations.native_run_stalled ?? 0).toBe(0);
|
|
expect(filtered.agentsWithStalledSlots).toEqual([]);
|
|
}, 60_000);
|
|
|
|
it("stops scheduling a task once it leaves eligibility", async () => {
|
|
const f = await seedCompany();
|
|
const done = await f.issue();
|
|
await f.run(done);
|
|
for (let index = 0; index < CONTINUATION_CAP; index++) {
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
|
|
entityType: "issue", entityId: done, details: {}, createdAt: new Date(Date.now() - (index + 1) * 8 * 60_000) });
|
|
}
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const deadlines: number[] = [];
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: done, onDeferredDeadline: at => deadlines.push(at) }))
|
|
.toMatchObject({ outcome: "deferred_cap" });
|
|
expect(deadlines).toHaveLength(1);
|
|
for (const change of [{ status: "done" }, { status: "cancelled" }, { assigneeAgentId: null, assigneeUserId: "person-1" }] as const) {
|
|
await db.update(issues).set({ status: "in_progress", assigneeAgentId: f.agentId, assigneeUserId: null, ...change }).where(eq(issues.id, done));
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: done, onDeferredDeadline: at => deadlines.push(at) })).toBeNull();
|
|
expect((await sweep.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(done);
|
|
}
|
|
expect(deadlines).toHaveLength(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("treats a stopped or blocked session goal as the person's move and only a running goal as pending", async () => {
|
|
const f = await seedCompany();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup });
|
|
const cases = [
|
|
{ goalStatus: "active", goalDesiredState: "active", expected: { clause: "pending_intent", intent: "session_goal" } },
|
|
{ goalStatus: "blocked", goalDesiredState: "paused", expected: { clause: "waiting", reason: "user_stop" } },
|
|
{ goalStatus: "active", goalDesiredState: "paused", expected: { clause: "waiting", reason: "user_stop" } },
|
|
];
|
|
for (const entry of cases) {
|
|
const issueId = await f.issue();
|
|
await f.run(issueId);
|
|
await db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: f.agentId, adapterType: "codex_local", taskKey: issueId,
|
|
goalStatus: entry.goalStatus, goalDesiredState: entry.goalDesiredState } as never);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "ok", verdict: entry.expected });
|
|
}
|
|
}, 60_000);
|
|
|
|
it("leaves a task whose run ended seconds ago, and asks to look again after the grace", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
const finishedAt = new Date(Date.now() - 2_000);
|
|
await f.run(issueId, { status: "failed", errorCode: "process_lost", finishedAt });
|
|
const queue = wakeQueue();
|
|
const deadlines: number[] = [];
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) }))
|
|
.toMatchObject({ outcome: "settling" });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(deadlines).toEqual([finishedAt.getTime() + 90_000]);
|
|
}, 60_000);
|
|
|
|
it("honours the per-task cap and the per-process repair budget; never charges the failure allowance", async () => {
|
|
const f = await seedCompany();
|
|
const capped = await f.issue();
|
|
await f.run(capped);
|
|
for (let index = 0; index < CONTINUATION_CAP; index++) {
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
|
|
entityType: "issue", entityId: capped, details: {}, createdAt: new Date(Date.now() - (index + 1) * 8 * 60_000) });
|
|
}
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
const deadlines: number[] = [];
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: capped, onDeferredDeadline: at => deadlines.push(at) }))
|
|
.toMatchObject({ outcome: "deferred_cap" });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(deadlines[0]).toBeGreaterThan(Date.now());
|
|
// The task stays as it was: never blocked for a person.
|
|
expect((await db.select().from(issues).where(eq(issues.id, capped)))[0].status).toBe("in_progress");
|
|
|
|
const many = [];
|
|
for (let index = 0; index < ASSIGNED_WORK_REPAIRS_PER_MINUTE + 2; index++) {
|
|
const id = await f.issue();
|
|
await f.run(id);
|
|
many.push(id);
|
|
}
|
|
const outcomes = [];
|
|
for (const id of many) outcomes.push((await sweep.checkIssue({ companyId: f.companyId, issueId: id }))?.outcome);
|
|
expect(outcomes.filter(outcome => outcome === "repaired")).toHaveLength(ASSIGNED_WORK_REPAIRS_PER_MINUTE);
|
|
expect(outcomes.filter(outcome => outcome === "deferred_rate_limit")).toHaveLength(2);
|
|
// Plain wakes: no retry-of link, so no failure-budget accounting.
|
|
const wake = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
expect(wake.every(row => !("retryOfRunId" in (row.payload ?? {})))).toBe(true);
|
|
}, 120_000);
|
|
|
|
it("counts the scheduler's transient retries of the same task against the one continuation cap", async () => {
|
|
const f = await seedCompany();
|
|
const issueId = await f.issue();
|
|
for (let index = 0; index < CONTINUATION_CAP - 2; index++) {
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
|
|
entityType: "issue", entityId: issueId, details: {}, createdAt: new Date(Date.now() - (50 - index * 10) * 60_000) });
|
|
}
|
|
const catalogFailure = (minutesAgo: number, attempt: number) => f.run(issueId, { status: "failed", errorCode: "repository_catalog_unavailable",
|
|
livenessState: null, scheduledRetryReason: "transient_failure", scheduledRetryAttempt: attempt,
|
|
resultJson: { errorFamily: "transient_upstream", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
createdAt: new Date(Date.now() - minutesAgo * 60_000), startedAt: new Date(Date.now() - minutesAgo * 60_000),
|
|
finishedAt: new Date(Date.now() - (minutesAgo - 1) * 60_000) });
|
|
await catalogFailure(9, 1);
|
|
await catalogFailure(8, 2);
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "deferred_cap" });
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("evaluates old-shaped rows: native awaiting reattach, a running run saved without its controller process, and stale blocks", async () => {
|
|
const f = await seedCompany();
|
|
const reattach = await f.issue();
|
|
const native = await f.run(reattach, { runtimeMode: "native", status: "failed", errorCode: "native_session_interrupted",
|
|
nativeIssueId: reattach });
|
|
await db.insert(nativeRunFinalizations).values({ runId: native.id, companyId: f.companyId, issueId: reattach, phase: "observed",
|
|
recoveryState: "awaiting_runner_reattach" });
|
|
const retryable = await f.issue();
|
|
const failedNative = await f.run(retryable, { runtimeMode: "native", status: "failed", errorCode: "saved_native_failure", nativeIssueId: retryable });
|
|
await db.insert(nativeRunFinalizations).values({ runId: failedNative.id, companyId: f.companyId, issueId: retryable,
|
|
phase: "retryable_failure", nextAttemptAt: new Date(Date.now() - 1_000) });
|
|
const unstamped = await f.issue();
|
|
await f.run(unstamped, { status: "running", errorCode: null, finishedAt: null, controllerBootId: randomUUID(), controllerProcess: null,
|
|
controllerLeaseExpiresAt: new Date(Date.now() - HOUR) });
|
|
const stale = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 8 * 24 * HOUR) });
|
|
await f.run(stale, { status: "failed", errorCode: "process_lost", finishedAt: new Date(Date.now() - 8 * 24 * HOUR) });
|
|
const queue = wakeQueue();
|
|
const settle = vi.fn(async () => undefined);
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
|
|
proveOwner: async () => ({ verdict: "gone", reason: "pid_dead" }) });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: reattach })).toMatchObject({ verdict: { intent: "native_recovery" } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: retryable })).toMatchObject({ verdict: { intent: "native_recovery" } });
|
|
// No recorded controller process: the owner is unprovable, so it stays live.
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: unstamped })).toMatchObject({ verdict: { clause: "live_run" } });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: stale })).toMatchObject({ outcome: "stale_reported" });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(settle).not.toHaveBeenCalled();
|
|
expect((await sweep.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(stale);
|
|
const report = await sweep.report(f.companyId);
|
|
expect(report).toMatchObject({ staleBlocked: 1, violationCount: 0 });
|
|
}, 60_000);
|
|
|
|
it("stays cheap at staging size: 800 open tasks and a week of runs at 2,000 a day", async () => {
|
|
const f = await seedCompany();
|
|
const issueRows = Array.from({ length: 800 }, (_, index) => ({ id: randomUUID(), companyId: f.companyId, title: `Load ${index}`,
|
|
status: index % 3 === 0 ? "blocked" : index % 3 === 1 ? "in_progress" : "todo", assigneeAgentId: f.agentId,
|
|
issueNumber: 10_000 + index, identifier: `L${f.companyId.slice(0, 6)}-${index}`,
|
|
blockedTransitionAt: index % 3 === 0 ? new Date(Date.now() - (index % 10) * 24 * HOUR) : null,
|
|
updatedAt: new Date(Date.now() - index * 60_000) }));
|
|
for (let index = 0; index < issueRows.length; index += 200) await db.insert(issues).values(issueRows.slice(index, index + 200));
|
|
const runs = Array.from({ length: 14_000 }, (_, index) => {
|
|
const issueId = issueRows[index % issueRows.length].id;
|
|
const at = new Date(Date.now() - (index / 14_000) * 7 * 24 * HOUR);
|
|
return { companyId: f.companyId, agentId: f.agentId, invocationSource: "automation", status: index < 40 ? "running" : "succeeded",
|
|
startedAt: at, finishedAt: index < 40 ? null : at, createdAt: at, contextSnapshot: { issueId, taskId: issueId },
|
|
controllerLeaseExpiresAt: index < 40 ? new Date(Date.now() + 60_000) : null, controllerBootId: index < 40 ? randomUUID() : null };
|
|
});
|
|
for (let index = 0; index < runs.length; index += 1_000) await db.insert(heartbeatRuns).values(runs.slice(index, index + 1_000));
|
|
await db.execute(sqlTag`analyze`);
|
|
const queue = wakeQueue();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
|
|
const time = async <T>(work: () => Promise<T>) => { const started = performance.now(); const value = await work();
|
|
return { value, ms: Math.round(performance.now() - started) }; };
|
|
await sweep.listSuspects({ companyId: f.companyId }, null, 100);
|
|
const suspects = await time(() => sweep.listSuspects({ companyId: f.companyId }, null, 100));
|
|
const one = await time(() => sweep.checkIssue({ companyId: f.companyId, issueId: suspects.value[0].id }));
|
|
const report = await time(() => sweep.report(f.companyId));
|
|
const plan = await db.execute<{ "QUERY PLAN": string }>(sqlTag`explain (analyze, buffers) select count(*) from issues i
|
|
where i.company_id = ${f.companyId} and i.status in ('todo','in_progress','blocked') and i.assignee_agent_id is not null
|
|
and not exists (select 1 from heartbeat_runs r where r.company_id = i.company_id and (r.context_snapshot->>'issueId') = i.id::text
|
|
and r.status in ('queued','running','scheduled_retry'))`);
|
|
const planText = (Array.isArray(plan) ? plan : (plan as { rows: Array<{ "QUERY PLAN": string }> }).rows).map(row => row["QUERY PLAN"]).join("\n");
|
|
console.info(JSON.stringify({ assignedWorkQueryCost: { suspectPageMs: suspects.ms, suspects: suspects.value.length,
|
|
checkOneIssueMs: one.ms, fullReportMs: report.ms, reportChecked: report.value.checked } }));
|
|
console.info(planText);
|
|
expect(planText).not.toMatch(/Seq Scan on heartbeat_runs/);
|
|
expect(suspects.ms).toBeLessThan(1_000);
|
|
expect(one.ms).toBeLessThan(1_000);
|
|
}, 240_000);
|
|
});
|
|
|