Files
PaperClipAI/server/src/__tests__/assigned-work-invariant.test.ts
T
Devin FoleyandPaperclip c41145e72a fix: wait on pending questions and quiet provider holds (#22363)
## Thinking Path

> - Paperclip manages agents and the tasks they work on.
> - Recovery should continue unfinished work when it is eligible.
> - An active unanswered question is a persisted reason to wait for a
person.
> - Treating a later Retry as an answer makes the recovery sweep
repeatedly request continuations that admission refuses.
> - Provider-limit holds also produce one cancellation toast per task,
even though provider work never started.
> - This PR aligns recovery with persisted interaction status and
presents these waits accurately.

## Linked Issues or Issue Description

Refs: #20466. This addresses the question blocker presentation and
automatic recovery loop; it preserves the existing
interaction-resolution policy.

Related work: #19420 schedules provider quota retries, and #21570
handles notices and reply ownership in agent chats. This change concerns
assigned-task reconciliation, task/inbox status, and per-run
cancellation toasts.

**What happened?**

The assigned-work sweep ignored pending interactions whenever a newer
run carried `explicitUserContinuation`. Continuation admission still
refused the pending question, so repeated sweeps recorded unsuccessful
continuation attempts. The task could meanwhile show an older provider
error as its current blocker. Separately, each pre-dispatch
provider-limit hold generated a cancellation toast, and the one-hour
fallback was described as a known reset and promised start time.

**Expected behavior**

Wait quietly while the question remains active under the shared
interaction eligibility rules, then react to the existing resolution
notification. Dismissed or historical ordinary questions remain
answerable without blocking work; governed requests keep their gates.
Show the current question on task and inbox surfaces. Keep provider
failure history, suppress cancellation toasts for expected provider
holds, and describe scheduled availability checks without promising the
provider will accept work.

**Steps to reproduce**

1. Leave an `ask_user_questions` interaction pending with
`supersedeOnUserComment: false`.
2. Create a later failed/interrupted explicit Retry and run
assigned-work reconciliation repeatedly.
3. Observe that the question remains pending while the sweep attempts
fresh continuations.
4. Separately, let an agent with recent provider rate-limit evidence
receive automatic task wakes without a reported reset time.

## What Changed

- Remove inferred question supersession from both assigned-work fact
loading and candidate selection. Use the existing active-interaction
predicate consistently in those reads, fresh-continuation admission, and
the task projection. Cover dismissed/historical ordinary questions,
retained governed gates, repeated checks, idle workers, and the existing
resolution notification.
- Add the `question` interruption action, load active pending question
IDs alongside approvals in one company-scoped query, and show that
blocker on tasks and inbox rows. Preserve a supported manual Retry after
a failed attempt, suppress duplicate actions for saved input, and keep
other current user gates authoritative. This preserves the existing
issue-detail SQL statement budget. Historical run failures remain
intact. Refresh the shared source SDK input pin.
- Suppress cancellation toasts specifically for cancelled
`provider_limit_wait` runs. Label provider retry deadlines as
availability rechecks, retain relative-time formatting, and record
successful scheduling as an informational lifecycle event.
- Document the wait contract in the continuation architecture guide.

## Verification

- Demonstrated failing regressions before the implementation:
continuation loop, task projection, task notice, cancellation toast,
provider deadline wording, and deadline formatting.
- Initial focused Vitest qualification: 642 tests passed across 14
files, using:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts
\

server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts
\
    server/src/services/recovery/assigned-work-invariant.test.ts \

ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx}
  ```
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- Review follow-up: 157 tests passed in `assigned-work-invariant`,
`task-interruption-query`, and `fresh-continuation-fallback`; 575 passed
in `task-interruption-query`, `assigned-work-recheck-idle`,
`task-interruption-presentation`, `issue-question-context`,
`issue-thread-interactions`, and `explicit-native-continuation` (the
query file overlaps). The strengthened final
dismissed/historical/governed projection regression also passed
separately.
- Final query-consolidation verification: 157 tests passed across five
files, including the unchanged 64-statement issue-detail census and
mixed approval/question batches:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts
\
    server/src/services/task-interruption-presentation.test.ts
  ```
- `pnpm --filter @paperclipai/server typecheck` — passed, including test
layers.
- Manual-recovery follow-up: 204 tests passed across seven
projection/recovery/notice suites, and all 104 Inbox tests passed. The
lost-runner integration still starts exactly one run from a manual Retry
and leaves the question unanswered. The saved-decision census verifies
one decision batch query for a 41-task list, with at most three
statements referencing interactions; the overall 64-statement
issue-detail limit remains unchanged.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/ui build` — passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` — passed.
- `pnpm verify` — exit 2: broader CI is required for changed global
inputs. Focused tests and package checks supply local evidence; broad CI
remains required.
- `git diff --check` and a local secret/instance-data scan — passed.
- Final CI on `1c16fe7c0d8`: [trusted CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842)
passed, including all server shards, native-server verification, build,
canary, e2e, release registry, and serialized suites. All PR checks are
complete, with no failures; the branch is conflict-free.
- Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no
actionable findings or unresolved review threads.

The focused tests cover genuine provider failures, live work and
terminal tasks, unchanged task locks/retry delivery, preservation of
historical failures, and question resolution waking the affected task
while other active pending questions remain quiet. Review regressions
also cover dismissed/historical ordinary questions retaining Retry and
governed requests retaining their gates. No live task was retried or
answered for validation.

## Risks

- An active pending question now consistently blocks automatic
assigned-work recovery. Shared interaction eligibility determines when
ordinary conversation input becomes deferred or historical; governed
requests retain their authority. Automatic recovery no longer infers
that a newer Retry resolved an active question.
- Manual Retry keeps its existing admission policy and recovery checks.
The question notice preserves that supported action after a failed
attempt without changing approval or credential authority; pending saved
input prevents duplicate actions.
- The shared interruption action gains `question`; the task and inbox
consumers are updated together. The pending-question query returns only
task IDs and uses the existing company/task/status index.
- Provider cooldown policy, quotas, credentials, routing, cancellation
history, and durable retry timing are unchanged. The actual provider can
still return a rate limit at the recheck.
- No migration is required. Existing large owner modules receive small
localized changes; no code file crosses 1,000 lines.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh
reasoning effort, with code execution and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 19:54:15 -08:00

882 lines
61 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq, sql as sqlTag } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { enableReattachPhase, joinReattachPhase, resetReattachPhaseForTests, settleReattachRun } from "../services/heartbeat/run-admission.js";
import {
activityLog, agents, agentTaskSessions, agentWakeupRequests, approvals, companies, createDb, heartbeatRuns, issueApprovals,
issueRecoveryActions, issueRelations, issues, issueThreadInteractions, nativeRunFinalizations,
} from "@paperclipai/db";
import { recoveryService } from "../services/recovery/service.js";
import { heartbeatService } from "../services/heartbeat.js";
import { assignedWorkSweep, ASSIGNED_WORK_CONTINUED_ACTION, ASSIGNED_WORK_REPAIRS_PER_MINUTE } from "../services/recovery/assigned-work-sweep.js";
import { CONTINUATION_CAP } from "../services/recovery/assigned-work-invariant.js";
import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
if (!recoveryPostgresSupport.supported) console.warn(`Skipping assigned-work invariant tests: ${recoveryPostgresSupport.reason}`);
const HOUR = 60 * 60_000;
/**
* The assigned-work invariant on real rows. Each stranded shape below was
* measured on staging on 2026-11-12; the existing stranded sweep leaves each
* of them with nothing pending, and the invariant sweep gives each exactly one
* ordinary continuation wake.
*/
describePostgres("assigned-work invariant sweep", () => {
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temporary = await startRecoveryPostgres("assigned-work-invariant-");
db = createDb(temporary.connectionString);
}, 240_000);
afterAll(async () => {
await db?.$client.end({ timeout: 0 });
await temporary?.cleanup();
});
/** A wake queue stand-in that records each wake as a durable pending row. */
function wakeQueue() {
const calls: Array<{ agentId: string; issueId: string; source: unknown }> = [];
const enqueueWakeup = vi.fn(async (agentId: string, opts?: { payload?: Record<string, unknown> | null; contextSnapshot?: Record<string, unknown>; reason?: string | null }) => {
const issueId = String(opts?.payload?.issueId);
const [issue] = await db.select({ companyId: issues.companyId }).from(issues).where(eq(issues.id, issueId));
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: issue.companyId, agentId, source: "automation",
triggerDetail: "system", reason: opts?.reason ?? null, payload: opts?.payload ?? null, status: "queued" }).returning();
calls.push({ agentId, issueId, source: opts?.contextSnapshot?.source });
return { id: wake.id } as never;
});
return { calls, enqueueWakeup };
}
async function seedCompany() {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Invariant fixture", issuePrefix: `I${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Builder", adapterType: "codex_local", status: "idle", adapterConfig: {} });
let number = 0;
async function issue(values: Partial<typeof issues.$inferInsert> = {}) {
const id = randomUUID();
number += 1;
await db.insert(issues).values({ id, companyId, title: `Task ${number}`, status: "in_progress", assigneeAgentId: agentId,
issueNumber: number, identifier: `I${companyId.slice(0, 7)}-${number}`, updatedAt: new Date(Date.now() - 3 * HOUR),
createdAt: new Date(Date.now() - 5 * HOUR), ...values });
return id;
}
async function run(issueId: string, values: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "interrupted",
errorCode: "server_shutdown_interrupted", startedAt: new Date(Date.now() - 4 * HOUR), finishedAt: new Date(Date.now() - 3 * HOUR),
livenessState: "needs_followup", contextSnapshot: { issueId, taskId: issueId }, createdAt: new Date(Date.now() - 4 * HOUR),
...values }).returning();
return row;
}
return { companyId, agentId, issue, run };
}
it("a deploy-interrupted in_progress task with nothing pending gets exactly one continuation; the stranded sweep leaves it", async () => {
const f = await seedCompany();
const issueId = await f.issue();
// The staging run: a person's Retry, stopped by a deploy's shutdown during
// setup, certified as no provider work, its hold already resolved.
await f.run(issueId, { resultJson: { stopReason: "interrupted", workspaceRestoreFailure: "restore_failed",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
contextSnapshot: { issueId, taskId: issueId, explicitUserContinuation: { previousRunId: randomUUID() } } });
const queue = wakeQueue();
// Production wiring: the bounded retry scheduler declines (not_scheduled).
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId, skipRelatedRecovery: true });
expect(queue.calls).toHaveLength(0);
const [stillStranded] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(stillStranded.status).toBe("in_progress");
const check = await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId });
// What staging shows for this shape: a Retry button that only the system can act on.
expect(check).toMatchObject({ outcome: "repaired", verdict: { kind: "violation", reason: "retry_offered" } });
expect(queue.calls).toEqual([{ agentId: f.agentId, issueId, source: "issue.assigned_work_invariant" }]);
const [activity] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
expect(activity.details).toMatchObject({ message: expect.stringContaining("Continued automatically") });
expect(String(activity.details?.message)).not.toMatch(/_|[0-9a-f]{8}-/);
// The next sweep sees the pending wake and leaves the task alone.
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent", intent: "queued_wake" } });
expect(queue.calls).toHaveLength(1);
}, 60_000);
it("a wake deferred behind a run that ended long ago is not a pending path; a fresh one still is", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId, { status: "failed", errorCode: "native_restart_recovery_blocked" });
const deferred = (createdAt: Date) => db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "automation", triggerDetail: "system", reason: "issue_execution_deferred", status: "deferred_issue_execution",
requestedByActorType: "user", requestedByActorId: "local-board", payload: { issueId }, createdAt, updatedAt: new Date() });
await deferred(new Date(Date.now() - 30 * 60_000));
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).toContain(issueId);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "repaired", verdict: { kind: "violation", reason: "no_pending_path" } });
const freshIssue = await f.issue();
await f.run(freshIssue, { status: "failed", errorCode: "native_restart_recovery_blocked" });
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
triggerDetail: "system", reason: "issue_execution_deferred", status: "deferred_issue_execution",
payload: { issueId: freshIssue }, createdAt: new Date(Date.now() - 60_000) });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).not.toContain(freshIssue);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: freshIssue }))
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent", intent: "queued_wake" } });
}, 60_000);
it("a wake claimed long ago that never became a run is not a pending path", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId, { status: "succeeded", errorCode: null, livenessState: null });
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
triggerDetail: "system", reason: "issue_children_completed", status: "claimed", payload: { issueId },
claimedAt: new Date(Date.now() - 15 * 60_000), createdAt: new Date(Date.now() - 40 * 60_000) });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).toContain(issueId);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ verdict: { kind: "violation", reason: "no_pending_path" } });
}, 60_000);
it("an in_progress task whose native run failed needing reconciliation (silently skipped today) gets one continuation", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation",
livenessState: "failed" });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId, skipRelatedRecovery: true });
expect(queue.calls).toHaveLength(0);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
expect(queue.calls).toHaveLength(1);
}, 60_000);
it("a task blocked after a legacy run's retries ran out, with no hold left, is restored and continued once", async () => {
// The shape the fresh-continuation fallback does not cover: retries spent
// on a restart-caused failure (controller lease lost), block written by
// the stranded sweep, and the hold already resolved.
const f = await seedCompany();
const blockedAt = new Date(Date.now() - 2 * HOUR);
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
const run = await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "stranded_assigned_issue",
status: "resolved", outcome: "blocked", ownerType: "board", cause: "stranded_assigned_issue",
fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "controller_lease_lost" },
nextAction: "Inspect the retry history.", attemptCount: 3, resolvedAt: blockedAt });
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(issueId);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "recovery_block" } });
expect(queue.calls).toHaveLength(1);
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("in_progress");
}, 60_000);
it("a recovery block whose task already has a queued run is lifted without a second run", async () => {
const f = await seedCompany();
const blockedAt = new Date(Date.now() - 20 * 60_000);
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
await f.run(issueId, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation",
livenessState: "failed", finishedAt: new Date(blockedAt.getTime() - 5_000) });
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "execution-recovery", action: "issue.updated",
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
reason: "native_continuation_requires_reconciliation" }, createdAt: blockedAt });
const successor = await f.run(issueId, { status: "queued", errorCode: null, livenessState: null, startedAt: null, finishedAt: null,
createdAt: new Date(blockedAt.getTime() + 19_000) });
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
triggerDetail: "system", reason: "issue_continuation_needed", status: "queued", runId: successor.id,
payload: { issueId } }).returning();
await db.update(heartbeatRuns).set({ wakeupRequestId: wake.id }).where(eq(heartbeatRuns.id, successor.id));
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(issueId);
expect(await recovery.assignedWork.report(f.companyId)).toMatchObject({ violations: { blocked_with_live_run: 1 } });
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "blocked_with_live_run", runId: successor.id } });
expect(queue.calls).toHaveLength(0);
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("todo");
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "ok", verdict: { clause: "live_run" } });
}, 60_000);
it("continues a recent board escalation of a temporary setup failure itself, and leaves old or lasting ones", async () => {
const f = await seedCompany();
async function escalated(input: { endedAgoMs: number; cause: string }) {
const blockedAt = new Date(Date.now() - input.endedAgoMs + 1_000);
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
const run = await f.run(issueId, { status: "failed", errorCode: "reusable_sandbox_resume_failed", livenessState: "failed",
error: `Failed to acquire lease: Reusable sandbox lease "x" could not be resumed (cause=${input.cause}); the lease was preserved and no replacement was created.`,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
startedAt: new Date(Date.now() - input.endedAgoMs - 60_000), finishedAt: new Date(Date.now() - input.endedAgoMs),
createdAt: new Date(Date.now() - input.endedAgoMs - 60_000) });
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId,
kind: "stranded_assigned_issue", status: "active", ownerType: "board", cause: "stranded_assigned_issue",
fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "reusable_sandbox_resume_failed",
routingPolicy: "board_escalation_no_takeover_v1" },
nextAction: "Board operator: inspect the evidence, repair the runtime if appropriate.", attemptCount: 1 }).returning();
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
entityType: "issue", entityId: issueId, details: { status: "blocked", previousStatus: "in_progress",
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
return { issueId, hold };
}
const recent = await escalated({ endedAgoMs: 2 * HOUR, cause: "transport_timeout" });
const old = await escalated({ endedAgoMs: 30 * HOUR, cause: "transport_timeout" });
const lasting = await escalated({ endedAgoMs: 2 * HOUR, cause: "provider_access_denied" });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
const suspects = (await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id);
expect(suspects).toContain(recent.issueId);
expect(suspects).not.toContain(old.issueId);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: recent.issueId }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "recovery_block" } });
expect(queue.calls).toEqual([{ agentId: f.agentId, issueId: recent.issueId, source: "issue.assigned_work_invariant" }]);
expect((await db.select().from(issues).where(eq(issues.id, recent.issueId)))[0].status).toBe("in_progress");
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, recent.hold.id)))[0])
.toMatchObject({ status: "resolved", outcome: "restored", ownerType: "system", resolutionNote: "stranded_transient_continued" });
for (const left of [old, lasting]) {
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: left.issueId }))
.toMatchObject({ outcome: "left_to_hold_owner" });
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, left.hold.id)))[0].status).toBe("active");
}
expect(queue.calls).toHaveLength(1);
}, 60_000);
it("leaves a task blocked under an active recovery hold to the hold's owner and reports it", async () => {
const f = await seedCompany();
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 2 * HOUR) });
const run = await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed",
startedAt: new Date(Date.now() - 31 * HOUR), finishedAt: new Date(Date.now() - 30 * HOUR), createdAt: new Date(Date.now() - 31 * HOUR) });
// Old-shaped row: a board-owned stranded action written by an older release.
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "stranded_assigned_issue",
status: "active", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: `source_scoped_recovery:${f.companyId}:${issueId}:stranded_assigned_issue`,
evidence: { previousStatus: "in_progress", latestRunId: run.id, latestRunErrorCode: "process_lost" },
nextAction: "Board operator: inspect the retry history, then explicitly retry the original owner, reassign, or intentionally resolve the task.",
attemptCount: 1 });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "left_to_hold_owner", verdict: { kind: "violation", reason: "recovery_block" } });
expect(queue.calls).toHaveLength(0);
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(issueId);
expect(await recovery.assignedWork.report(f.companyId)).toMatchObject({ violationCount: 1, heldByRecoveryOwner: 1 });
}, 60_000);
it("continues a refused Retry only after its pending question is actually resolved", async () => {
// Nine staging tasks: a person's Retry, stopped by a deploy during setup,
// certified as no provider work; the scheduler then refused with
// continuation_user_authorization_missing on every sweep. One of them
// also had a question the agent asked before the person pressed Retry.
const f = await seedCompany();
const plain = await f.issue();
const withQuestion = await f.issue();
for (const issueId of [plain, withQuestion]) {
await f.run(issueId, { resultJson: { stopReason: "interrupted", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
contextSnapshot: { issueId, taskId: issueId, explicitUserContinuation: { previousRunId: randomUUID() } },
createdAt: new Date(Date.now() - 4 * HOUR) });
}
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: withQuestion, kind: "ask_user_questions",
status: "pending", continuationPolicy: "wake_assignee", payload: { supersedeOnUserComment: false }, createdAt: new Date(Date.now() - 12 * HOUR) } as never);
const queue = wakeQueue();
// On staging the refusal also left the sweep's escalation without effect
// (its stale-snapshot guard): the tasks stayed in_progress with nothing
// pending. Model that by changing the task under the sweep's snapshot.
const scheduleRecoveryRetry = vi.fn(async (runId: string, outcome?: { continuationRefused?: boolean }) => {
if (outcome) outcome.continuationRefused = true;
const [run] = await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
await db.update(issues).set({ updatedAt: new Date(Date.now() - 2 * HOUR) }).where(eq(issues.id, String(run.context?.issueId)));
return null;
});
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry });
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, skipRelatedRecovery: true });
expect(queue.calls).toHaveLength(0);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: plain }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "retry_offered" } });
for (let pass = 0; pass < 3; pass += 1) {
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: withQuestion }))
.toMatchObject({ outcome: "ok", verdict: { clause: "waiting", reason: "question" } });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id))
.not.toContain(withQuestion);
}
expect(queue.calls.map(call => call.issueId)).toEqual([plain]);
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, withQuestion),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)))).toHaveLength(0);
await db.update(issueThreadInteractions).set({ status: "answered" }).where(eq(issueThreadInteractions.issueId, withQuestion));
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id))
.toContain(withQuestion);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: withQuestion }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "retry_offered" } });
expect(queue.calls.map(call => call.issueId).sort()).toEqual([plain, withQuestion].sort());
}, 60_000);
it("an unknown block reason is repaired, not treated as a reason to wait", async () => {
const f = await seedCompany();
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
await f.run(issueId, { status: "failed", errorCode: "something_new", livenessState: "failed" });
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "repaired", verdict: { reason: "unknown_block" } });
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("todo");
expect(queue.calls).toHaveLength(1);
}, 60_000);
it("continues past a dismissed ordinary question while retaining a governed question's gate", async () => {
const f = await seedCompany();
const ordinary = await f.issue(), governed = await f.issue();
for (const issueId of [ordinary, governed]) {
await f.run(issueId);
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId, kind: "ask_user_questions",
status: "pending", dismissedAt: new Date(), payload: { version: 1, questions: [],
...(issueId === governed ? { toolAction: {} } : {}) } } as never);
}
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const suspects = (await sweep.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id);
expect(suspects).toContain(ordinary);
expect(suspects).not.toContain(governed);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: ordinary })).toMatchObject({ outcome: "repaired" });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: governed }))
.toMatchObject({ outcome: "ok", verdict: { clause: "waiting", reason: "question" } });
expect(queue.calls.map(call => call.issueId)).toEqual([ordinary]);
}, 60_000);
it("two servers sweeping the same violation at once create one wake", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId);
const queue = wakeQueue();
const serverA = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const serverB = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const results = await Promise.all([serverA.checkIssue({ companyId: f.companyId, issueId }), serverB.checkIssue({ companyId: f.companyId, issueId })]);
expect(queue.calls).toHaveLength(1);
expect(results.map(result => result?.outcome).sort()).toEqual(["repaired", expect.stringMatching(/changed|deferred_rate_limit/)].sort());
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId), eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))))
.toHaveLength(1);
}, 60_000);
it("leaves a run owned by another live server alone, and settles one whose owner is proven gone", async () => {
const f = await seedCompany();
const otherBoot = randomUUID();
const controllerProcess = { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() };
// Deploy overlap: the other server holds the lease.
const leased = await f.issue();
await f.run(leased, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
controllerLeaseExpiresAt: new Date(Date.now() + 60_000) });
// Lease lapsed, but the owner proof says its server is still alive.
const lapsedAlive = await f.issue();
await f.run(lapsedAlive, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
const queue = wakeQueue();
const alive = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup,
proveOwner: async () => ({ verdict: "alive", reason: "controller_lease_held" }) });
expect(await alive.checkIssue({ companyId: f.companyId, issueId: leased })).toMatchObject({ verdict: { clause: "live_run" } });
expect(await alive.checkIssue({ companyId: f.companyId, issueId: lapsedAlive })).toMatchObject({ verdict: { clause: "live_run" } });
expect(queue.calls).toHaveLength(0);
const settle = vi.fn(async () => undefined);
const gone = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }) });
expect(await gone.checkIssue({ companyId: f.companyId, issueId: leased })).toMatchObject({ verdict: { clause: "live_run" } });
const goneCheck = await gone.checkIssue({ companyId: f.companyId, issueId: lapsedAlive });
expect(goneCheck)
.toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone" } });
expect(settle).toHaveBeenCalledTimes(1);
expect(queue.calls).toHaveLength(0);
}, 60_000);
it("takes no action while handed-over live runs are being reached, and acts once the phase ends", async () => {
const f = await seedCompany();
const otherBoot = randomUUID();
const controllerProcess = { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() };
const lapsed = await f.issue();
await f.run(lapsed, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot, controllerProcess,
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
const queue = wakeQueue();
const settle = vi.fn(async () => undefined);
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }) });
try {
enableReattachPhase(180_000);
joinReattachPhase([{ runId: "handed-over-live-run", reason: "handed_over", claimableAt: Date.now() }]);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: lapsed }))
.toMatchObject({ outcome: "deferred_reattach_phase", verdict: { reason: "run_owner_gone" } });
expect(settle).not.toHaveBeenCalled();
expect(queue.calls).toHaveLength(0);
settleReattachRun("handed-over-live-run", "reached");
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: lapsed }))
.toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone" } });
expect(settle).toHaveBeenCalledTimes(1);
} finally {
resetReattachPhaseForTests();
}
}, 60_000);
it("leaves tasks waiting on an approval, a question, a dependency or a declared unblock action", async () => {
const f = await seedCompany();
const approvalTask = await f.issue();
const [approval] = await db.insert(approvals).values({ companyId: f.companyId, type: "plan", status: "pending", payload: {} } as never).returning();
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: approvalTask, approvalId: approval.id } as never);
const questionTask = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
await f.run(questionTask, { runtimeMode: "native", status: "failed", errorCode: "native_continuation_requires_reconciliation" });
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: questionTask, kind: "request_confirmation",
status: "pending", continuationPolicy: "wake_assignee", payload: {} } as never);
const dependent = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR) });
const blocker = await f.issue({ status: "todo", assigneeAgentId: null });
await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blocker, relatedIssueId: dependent, type: "blocks" } as never);
const declared = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - HOUR),
unblockDescriptor: { owner: "board", action: "Share the deploy key for the staging project." } });
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: approvalTask })).toMatchObject({ verdict: { reason: "approval" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: questionTask })).toMatchObject({ verdict: { reason: "question" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: dependent })).toMatchObject({ verdict: { reason: "dependency" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: declared })).toMatchObject({ verdict: { reason: "assignee_declared_block" } });
expect(queue.calls).toHaveLength(0);
// The indexed pre-filter leaves all four out of the sweep entirely.
const suspects = await sweep.listSuspects({ companyId: f.companyId }, null, 100);
expect(suspects.map(row => row.id)).not.toEqual(expect.arrayContaining([approvalTask]));
for (const id of [approvalTask, questionTask, dependent, declared]) expect(suspects.map(row => row.id)).not.toContain(id);
}, 60_000);
it("a refused or later-declined continuation stays a violation, says so, and is tried again under the cap", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId);
let admit = false;
const enqueueWakeup = vi.fn(async (agentId: string, opts?: { payload?: Record<string, unknown> | null; reason?: string | null }) => {
if (!admit) return null;
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId, source: "automation",
triggerDetail: "system", reason: opts?.reason ?? null, payload: opts?.payload ?? null, status: "queued" }).returning();
return { id: wake.id } as never;
});
let clock = new Date();
const sweep = assignedWorkSweep(db, { enqueueWakeup, now: () => clock });
const deadlines: number[] = [];
// Admission refuses: the entry says nothing started, and a re-check is due.
expect(await sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) }))
.toMatchObject({ outcome: "enqueue_refused" });
const activity = async () => db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
expect((await activity())[0].details).toMatchObject({ outcome: "not_started", message: expect.stringContaining("could not start") });
expect(deadlines).toHaveLength(1);
// Too soon: spaced, not repaired and not reported as fine.
clock = new Date(clock.getTime() + 60_000);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "waiting_unchanged", verdict: { kind: "violation" } });
// After the spacing it is admitted; the queue later declines the claim
// (for example behind the old run's pending cleanup) and the wake is skipped.
admit = true;
clock = new Date(clock.getTime() + 2 * 60_000);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
expect((await activity()).map(row => row.details?.outcome).sort()).toEqual(["not_started", "queued"]);
await db.update(agentWakeupRequests).set({ status: "skipped" }).where(eq(agentWakeupRequests.companyId, f.companyId));
clock = new Date(clock.getTime() + 5 * 60_000);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
expect(enqueueWakeup).toHaveBeenCalledTimes(3);
}, 60_000);
it("with the real wake queue, a hold reached by the sweep's wake becomes one pending fresh continuation", async () => {
// The task's last run left a legacy reconciliation hold. The sweep's
// ordinary continuation wake reaches the fresh-continuation fallback,
// which settles the hold and records one pending continuation (owned by
// the execution-continuation worker). The next sweep adds no attempt.
const f = await seedCompany();
const issueId = await f.issue();
const run = await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed" });
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
status: "active", ownerType: "board", cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${run.id}`,
evidence: { runId: run.id, originalFailureCode: "process_lost" }, nextAction: "Reconcile the stopped run.", attemptCount: 1 });
const heartbeat = heartbeatService(db);
try {
// The wake became the continuation, so the sweep counts it as repaired.
const first = await heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId });
expect(first).toMatchObject({ outcome: "repaired", verdict: { kind: "violation" } });
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sqlTag`${heartbeatRuns.status} in ('queued', 'running', 'scheduled_retry')`));
expect(queued).toHaveLength(0);
const pending = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
sqlTag`${agentWakeupRequests.status} in ('queued', 'deferred_issue_execution', 'claimed')`));
expect(pending).toHaveLength(0);
const [entry] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
expect(entry.details).toMatchObject({ outcome: "queued", continuedFromSavedWork: "true" });
const [record] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
sqlTag`${issueRecoveryActions.evidence}->'freshContinuation' is not null`));
expect(record!.evidence).toMatchObject({ continuationDelivery: "pending" });
// A second sweep right away adds no attempt.
expect(await heartbeat.checkAssignedWorkInvariant({ companyId: f.companyId, issueId }))
.not.toMatchObject({ outcome: "repaired" });
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)))).toHaveLength(1);
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
it("routine touches on a stranded task (a comment, a read, a label) do not postpone its repair", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId);
await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, issueId));
const queue = wakeQueue();
expect(await assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup }).checkIssue({ companyId: f.companyId, issueId }))
.toMatchObject({ outcome: "repaired" });
expect(queue.calls).toHaveLength(1);
}, 60_000);
it("restores a block made through the normal update route to its recorded status, and a review block back to review without a wake", async () => {
const f = await seedCompany();
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const blockedAt = new Date(Date.now() - 2 * HOUR);
const cases = [["in_progress", "in_progress", 1], ["in_review", "in_review", 0]] as const;
for (const [before, expected, wakes] of cases) {
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
await f.run(issueId, { status: "failed", errorCode: "process_lost", livenessState: "failed" });
// The shape the normal issue update route writes.
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
entityType: "issue", entityId: issueId, details: { status: "blocked", _previous: { status: before } }, createdAt: blockedAt });
const callsBefore = queue.calls.length;
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "repaired" });
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe(expected);
expect(queue.calls.length - callsBefore).toBe(wakes);
}
}, 60_000);
it("puts a lifted block back when admission refuses the continuation", async () => {
const f = await seedCompany();
const blockedAt = new Date(Date.now() - 2 * HOUR);
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: blockedAt });
await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
const notifiedAt = new Date(Date.now() - HOUR);
const descriptor = { owner: { agentId: f.agentId }, action: "Retry the build once the cache is warm." };
await db.update(issues).set({ unblockDescriptor: descriptor as never, blockedOwnerNotifiedAt: notifiedAt }).where(eq(issues.id, issueId));
const sweep = assignedWorkSweep(db, { enqueueWakeup: async () => null });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "enqueue_refused" });
const [after] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(after.status).toBe("blocked");
expect(after.blockedTransitionAt?.getTime()).toBe(blockedAt.getTime());
expect(after.unblockDescriptor).toEqual(descriptor);
expect(after.blockedOwnerNotifiedAt?.getTime()).toBe(notifiedAt.getTime());
}, 60_000);
it("re-runs the predicate under the row lock: a question asked after the first read stops the repair", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId);
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
// The first read sees a violation; the person's question lands just before
// the claim's transaction takes the row lock.
const original = db.transaction.bind(db);
const spy = vi.spyOn(db, "transaction").mockImplementationOnce((async (work: never, config?: never) => {
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId, kind: "ask_user_questions",
status: "pending", continuationPolicy: "wake_assignee", payload: {} } as never);
return original(work, config);
}) as never);
try {
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "changed", verdict: { kind: "violation" } });
} finally { spy.mockRestore(); }
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)))).toHaveLength(0);
expect(queue.calls).toHaveLength(0);
}, 60_000);
it("reports only tasks the caller may read; the sweep still evaluates private tasks", async () => {
const f = await seedCompany();
const open = await f.issue();
await f.run(open);
const hidden = await f.issue();
await f.run(hidden);
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
// A caller whose read condition excludes the private task.
const report = await sweep.report(f.companyId, { readCondition: sqlTag<boolean>`${issues.id} <> ${hidden}` });
expect(report).toMatchObject({ checked: 1, violationCount: 1 });
expect(report.sample.map(row => row.identifier)).not.toContain((await db.select().from(issues).where(eq(issues.id, hidden)))[0].identifier);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: hidden })).toMatchObject({ outcome: "repaired" });
}, 60_000);
it("completes a claim on a one-connection pool: the locked re-check reads through the transaction", async () => {
const single = createDb(temporary.connectionString, { maxConnections: 1 });
try {
const f = await seedCompany();
const issueId = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 2 * HOUR) });
await f.run(issueId, { status: "failed", errorCode: "controller_lease_lost", livenessState: "failed" });
const queue = wakeQueue();
const sweep = assignedWorkSweep(single, { enqueueWakeup: queue.enqueueWakeup });
const result = await Promise.race([sweep.checkIssue({ companyId: f.companyId, issueId }),
new Promise(resolve => setTimeout(() => resolve("deadlocked"), 20_000))]);
expect(result).toMatchObject({ outcome: "repaired" });
expect(queue.calls).toHaveLength(1);
} finally { await single.$client.end({ timeout: 0 }); }
}, 60_000);
it("says when the report is capped, and that the violation count is then a lower bound", async () => {
const f = await seedCompany();
const issueId = await f.issue();
await f.run(issueId);
const report = await assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup }).report(f.companyId);
expect(report).toMatchObject({ checked: 1, openTasks: 1, truncated: false, violationCountIsLowerBound: false, violationCount: 1 });
}, 60_000);
it("a running run whose owner is proven gone is settled first, then the task gets its continuation within the bound", async () => {
const f = await seedCompany();
const issueId = await f.issue();
const otherBoot = randomUUID();
const run = await f.run(issueId, { status: "running", errorCode: null, finishedAt: null, controllerBootId: otherBoot,
controllerProcess: { bootId: otherBoot, pid: 4242, hostHash: "host-b", processStartedAt: new Date(Date.now() - HOUR).toISOString() },
controllerLeaseExpiresAt: new Date(Date.now() - 60_000) });
const queue = wakeQueue();
// Heartbeat's lost-process settle path ends the run, as it does in production.
const settle = vi.fn(async ({ runId }: { runId: string }) => {
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
});
let clock = new Date();
const started = clock.getTime();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
proveOwner: async () => ({ verdict: "gone", reason: "controller_stopped" }), now: () => clock });
const deadlines: number[] = [];
const pass = () => sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) });
// Pass 1: the owner is proven gone outside any lock; the run goes to the settle path, no claim is tried.
expect(await pass()).toMatchObject({ outcome: "settled_lost_run", verdict: { reason: "run_owner_gone", runId: run.id } });
expect(settle).toHaveBeenCalledTimes(1);
expect(queue.calls).toHaveLength(0);
// Pass 2 at its deadline: the run has ended and the task is repaired.
clock = new Date(deadlines.at(-1)! + 1_000);
expect(await pass()).toMatchObject({ outcome: "repaired" });
expect(queue.calls).toHaveLength(1);
expect(clock.getTime() - started).toBeLessThan(5 * 60_000);
}, 60_000);
it("does not count a native run left running in awaiting evidence with no renewal as live, and reports it with its agent's held slot", async () => {
// The graceful hand-off repro: the new server's startup recovery parked
// the run in awaiting evidence; nothing renews or settles it.
const f = await seedCompany();
const stuck = await f.issue();
const stuckRun = await f.run(stuck, { runtimeMode: "native", status: "running", errorCode: null, finishedAt: null, nativeIssueId: stuck });
const old = new Date(Date.now() - 10 * 60_000);
await db.insert(nativeRunFinalizations).values({ runId: stuckRun.id, companyId: f.companyId, issueId: stuck, phase: "observed",
recoveryState: "awaiting_evidence", leaseExpiresAt: old, updatedAt: old });
// A recent claim is still a live path.
const fresh = await f.issue();
const freshRun = await f.run(fresh, { runtimeMode: "native", status: "running", errorCode: null, finishedAt: null, nativeIssueId: fresh });
await db.insert(nativeRunFinalizations).values({ runId: freshRun.id, companyId: f.companyId, issueId: fresh, phase: "observed",
recoveryState: "awaiting_evidence", leaseExpiresAt: new Date(Date.now() + 60_000) });
// Queued work for the same agent that cannot dispatch behind the held slot.
const waiting = await f.issue({ status: "todo" });
await f.run(waiting, { status: "queued", errorCode: null, finishedAt: null, startedAt: null });
const queue = wakeQueue();
const settle = vi.fn(async () => undefined);
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: stuck }))
.toMatchObject({ outcome: "left_to_native_recovery", verdict: { kind: "violation", reason: "native_run_stalled", runId: stuckRun.id } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: fresh })).toMatchObject({ verdict: { clause: "live_run" } });
expect(queue.calls).toHaveLength(0);
expect(settle).not.toHaveBeenCalled();
const report = await sweep.report(f.companyId);
expect(report.violations).toMatchObject({ native_run_stalled: 1 });
expect(report.agentsWithStalledSlots).toEqual([{ agentName: "Builder", stalledRuns: 1, queuedRuns: 1 }]);
// A caller who cannot read those tasks sees none of their runs counted.
const hiddenIds = [stuck, waiting];
const filtered = await sweep.report(f.companyId, { readCondition: sqlTag<boolean>`${issues.id} not in (${sqlTag.join(hiddenIds.map(id => sqlTag`${id}::uuid`), sqlTag`, `)})` });
expect(filtered.violations.native_run_stalled ?? 0).toBe(0);
expect(filtered.agentsWithStalledSlots).toEqual([]);
}, 60_000);
it("stops scheduling a task once it leaves eligibility", async () => {
const f = await seedCompany();
const done = await f.issue();
await f.run(done);
for (let index = 0; index < CONTINUATION_CAP; index++) {
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
entityType: "issue", entityId: done, details: {}, createdAt: new Date(Date.now() - (index + 1) * 8 * 60_000) });
}
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const deadlines: number[] = [];
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: done, onDeferredDeadline: at => deadlines.push(at) }))
.toMatchObject({ outcome: "deferred_cap" });
expect(deadlines).toHaveLength(1);
for (const change of [{ status: "done" }, { status: "cancelled" }, { assigneeAgentId: null, assigneeUserId: "person-1" }] as const) {
await db.update(issues).set({ status: "in_progress", assigneeAgentId: f.agentId, assigneeUserId: null, ...change }).where(eq(issues.id, done));
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: done, onDeferredDeadline: at => deadlines.push(at) })).toBeNull();
expect((await sweep.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(done);
}
expect(deadlines).toHaveLength(1);
expect(queue.calls).toHaveLength(0);
}, 60_000);
it("treats a stopped or blocked session goal as the person's move and only a running goal as pending", async () => {
const f = await seedCompany();
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup });
const cases = [
{ goalStatus: "active", goalDesiredState: "active", expected: { clause: "pending_intent", intent: "session_goal" } },
{ goalStatus: "blocked", goalDesiredState: "paused", expected: { clause: "waiting", reason: "user_stop" } },
{ goalStatus: "active", goalDesiredState: "paused", expected: { clause: "waiting", reason: "user_stop" } },
];
for (const entry of cases) {
const issueId = await f.issue();
await f.run(issueId);
await db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: f.agentId, adapterType: "codex_local", taskKey: issueId,
goalStatus: entry.goalStatus, goalDesiredState: entry.goalDesiredState } as never);
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "ok", verdict: entry.expected });
}
}, 60_000);
it("leaves a task whose run ended seconds ago, and asks to look again after the grace", async () => {
const f = await seedCompany();
const issueId = await f.issue();
const finishedAt = new Date(Date.now() - 2_000);
await f.run(issueId, { status: "failed", errorCode: "process_lost", finishedAt });
const queue = wakeQueue();
const deadlines: number[] = [];
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId, onDeferredDeadline: at => deadlines.push(at) }))
.toMatchObject({ outcome: "settling" });
expect(queue.calls).toHaveLength(0);
expect(deadlines).toEqual([finishedAt.getTime() + 90_000]);
}, 60_000);
it("honours the per-task cap and the per-process repair budget; never charges the failure allowance", async () => {
const f = await seedCompany();
const capped = await f.issue();
await f.run(capped);
for (let index = 0; index < CONTINUATION_CAP; index++) {
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
entityType: "issue", entityId: capped, details: {}, createdAt: new Date(Date.now() - (index + 1) * 8 * 60_000) });
}
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const deadlines: number[] = [];
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: capped, onDeferredDeadline: at => deadlines.push(at) }))
.toMatchObject({ outcome: "deferred_cap" });
expect(queue.calls).toHaveLength(0);
expect(deadlines[0]).toBeGreaterThan(Date.now());
// The task stays as it was: never blocked for a person.
expect((await db.select().from(issues).where(eq(issues.id, capped)))[0].status).toBe("in_progress");
const many = [];
for (let index = 0; index < ASSIGNED_WORK_REPAIRS_PER_MINUTE + 2; index++) {
const id = await f.issue();
await f.run(id);
many.push(id);
}
const outcomes = [];
for (const id of many) outcomes.push((await sweep.checkIssue({ companyId: f.companyId, issueId: id }))?.outcome);
expect(outcomes.filter(outcome => outcome === "repaired")).toHaveLength(ASSIGNED_WORK_REPAIRS_PER_MINUTE);
expect(outcomes.filter(outcome => outcome === "deferred_rate_limit")).toHaveLength(2);
// Plain wakes: no retry-of link, so no failure-budget accounting.
const wake = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
expect(wake.every(row => !("retryOfRunId" in (row.payload ?? {})))).toBe(true);
}, 120_000);
it("counts the scheduler's transient retries of the same task against the one continuation cap", async () => {
const f = await seedCompany();
const issueId = await f.issue();
for (let index = 0; index < CONTINUATION_CAP - 2; index++) {
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
entityType: "issue", entityId: issueId, details: {}, createdAt: new Date(Date.now() - (50 - index * 10) * 60_000) });
}
const catalogFailure = (minutesAgo: number, attempt: number) => f.run(issueId, { status: "failed", errorCode: "repository_catalog_unavailable",
livenessState: null, scheduledRetryReason: "transient_failure", scheduledRetryAttempt: attempt,
resultJson: { errorFamily: "transient_upstream", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
createdAt: new Date(Date.now() - minutesAgo * 60_000), startedAt: new Date(Date.now() - minutesAgo * 60_000),
finishedAt: new Date(Date.now() - (minutesAgo - 1) * 60_000) });
await catalogFailure(9, 1);
await catalogFailure(8, 2);
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId })).toMatchObject({ outcome: "deferred_cap" });
expect(queue.calls).toHaveLength(0);
}, 60_000);
it("evaluates old-shaped rows: native awaiting reattach, a running run saved without its controller process, and stale blocks", async () => {
const f = await seedCompany();
const reattach = await f.issue();
const native = await f.run(reattach, { runtimeMode: "native", status: "failed", errorCode: "native_session_interrupted",
nativeIssueId: reattach });
await db.insert(nativeRunFinalizations).values({ runId: native.id, companyId: f.companyId, issueId: reattach, phase: "observed",
recoveryState: "awaiting_runner_reattach" });
const retryable = await f.issue();
const failedNative = await f.run(retryable, { runtimeMode: "native", status: "failed", errorCode: "saved_native_failure", nativeIssueId: retryable });
await db.insert(nativeRunFinalizations).values({ runId: failedNative.id, companyId: f.companyId, issueId: retryable,
phase: "retryable_failure", nextAttemptAt: new Date(Date.now() - 1_000) });
const unstamped = await f.issue();
await f.run(unstamped, { status: "running", errorCode: null, finishedAt: null, controllerBootId: randomUUID(), controllerProcess: null,
controllerLeaseExpiresAt: new Date(Date.now() - HOUR) });
const stale = await f.issue({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 8 * 24 * HOUR) });
await f.run(stale, { status: "failed", errorCode: "process_lost", finishedAt: new Date(Date.now() - 8 * 24 * HOUR) });
const queue = wakeQueue();
const settle = vi.fn(async () => undefined);
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, settleLostRunProcess: settle,
proveOwner: async () => ({ verdict: "gone", reason: "pid_dead" }) });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: reattach })).toMatchObject({ verdict: { intent: "native_recovery" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: retryable })).toMatchObject({ verdict: { intent: "native_recovery" } });
// No recorded controller process: the owner is unprovable, so it stays live.
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: unstamped })).toMatchObject({ verdict: { clause: "live_run" } });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: stale })).toMatchObject({ outcome: "stale_reported" });
expect(queue.calls).toHaveLength(0);
expect(settle).not.toHaveBeenCalled();
expect((await sweep.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).not.toContain(stale);
const report = await sweep.report(f.companyId);
expect(report).toMatchObject({ staleBlocked: 1, violationCount: 0 });
}, 60_000);
it("stays cheap at staging size: 800 open tasks and a week of runs at 2,000 a day", async () => {
const f = await seedCompany();
const issueRows = Array.from({ length: 800 }, (_, index) => ({ id: randomUUID(), companyId: f.companyId, title: `Load ${index}`,
status: index % 3 === 0 ? "blocked" : index % 3 === 1 ? "in_progress" : "todo", assigneeAgentId: f.agentId,
issueNumber: 10_000 + index, identifier: `L${f.companyId.slice(0, 6)}-${index}`,
blockedTransitionAt: index % 3 === 0 ? new Date(Date.now() - (index % 10) * 24 * HOUR) : null,
updatedAt: new Date(Date.now() - index * 60_000) }));
for (let index = 0; index < issueRows.length; index += 200) await db.insert(issues).values(issueRows.slice(index, index + 200));
const runs = Array.from({ length: 14_000 }, (_, index) => {
const issueId = issueRows[index % issueRows.length].id;
const at = new Date(Date.now() - (index / 14_000) * 7 * 24 * HOUR);
return { companyId: f.companyId, agentId: f.agentId, invocationSource: "automation", status: index < 40 ? "running" : "succeeded",
startedAt: at, finishedAt: index < 40 ? null : at, createdAt: at, contextSnapshot: { issueId, taskId: issueId },
controllerLeaseExpiresAt: index < 40 ? new Date(Date.now() + 60_000) : null, controllerBootId: index < 40 ? randomUUID() : null };
});
for (let index = 0; index < runs.length; index += 1_000) await db.insert(heartbeatRuns).values(runs.slice(index, index + 1_000));
await db.execute(sqlTag`analyze`);
const queue = wakeQueue();
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup });
const time = async <T>(work: () => Promise<T>) => { const started = performance.now(); const value = await work();
return { value, ms: Math.round(performance.now() - started) }; };
await sweep.listSuspects({ companyId: f.companyId }, null, 100);
const suspects = await time(() => sweep.listSuspects({ companyId: f.companyId }, null, 100));
const one = await time(() => sweep.checkIssue({ companyId: f.companyId, issueId: suspects.value[0].id }));
const report = await time(() => sweep.report(f.companyId));
const plan = await db.execute<{ "QUERY PLAN": string }>(sqlTag`explain (analyze, buffers) select count(*) from issues i
where i.company_id = ${f.companyId} and i.status in ('todo','in_progress','blocked') and i.assignee_agent_id is not null
and not exists (select 1 from heartbeat_runs r where r.company_id = i.company_id and (r.context_snapshot->>'issueId') = i.id::text
and r.status in ('queued','running','scheduled_retry'))`);
const planText = (Array.isArray(plan) ? plan : (plan as { rows: Array<{ "QUERY PLAN": string }> }).rows).map(row => row["QUERY PLAN"]).join("\n");
console.info(JSON.stringify({ assignedWorkQueryCost: { suspectPageMs: suspects.ms, suspects: suspects.value.length,
checkOneIssueMs: one.ms, fullReportMs: report.ms, reportChecked: report.value.checked } }));
console.info(planText);
expect(planText).not.toMatch(/Seq Scan on heartbeat_runs/);
expect(suspects.ms).toBeLessThan(1_000);
expect(one.ms).toBeLessThan(1_000);
}, 240_000);
});