mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
fix(recovery): wait for provider quota resets (#9635)
## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies and keep assigned work moving safely. > - The recovery subsystem decides whether a failed agent run should retry, wait, block for configuration, or escalate to another owner. > - Provider usage-limit failures currently arrive as generic `adapter_failed` results, so stranded-work reconciliation can create takeover recovery even when the provider states that capacity will reset later. > - Credential and model lookup failures are also configuration problems, not evidence that another agent should take over the task. > - This pull request classifies those failure families at recovery time and persists the classification on the run. > - Quota failures now schedule a monitor for the original assignee at the parsed reset time, or after a bounded default backoff when no reset time is available. > - The benefit is that transient provider capacity waits no longer wake recovery owners, while configuration failures stop with an actionable classification. ## Linked Issues or Issue Description No public GitHub issue exists for this exact change. **What happened?** When an assigned issue's latest run failed with a provider usage-limit message such as "try again at 12:00 AM (UTC)," recovery treated the run as generic `adapter_failed` work and could create a takeover action. Missing credentials and `model_not_found` failures followed the same generic path. **Expected behavior:** Provider quota failures should keep the original assignee and schedule a monitor for the reset time, without creating recovery work or immediately waking another owner. Missing credentials and model lookup failures should be classified as `configuration_incomplete` and blocked with the configuration fix recorded. **Steps to reproduce:** 1. Assign and start an issue for an agent. 2. Record a failed heartbeat run with `errorCode: adapter_failed` and a provider quota/reset message. 3. Run stranded assigned-issue reconciliation. 4. Observe that the old behavior routes the issue through generic recovery instead of waiting for provider capacity. Reproduced on `master` at `9af96461d`. This is a core recovery bug, not adapter-specific, and applies to built-from-source deployments with either embedded PGlite or Postgres. Related work checked: #9288 adds adapter-side Claude provider-limit classification; #5392 suppresses some recovery creation for quota-class errors; #9634 is a broader recovery-routing change with overlapping provider-quota behavior. This PR is the narrow recovery-service fix with focused parsed-reset, fallback-backoff, zero-takeover, and configuration-failure coverage. ## What Changed - Added conservative recovery-time classification for provider quota, missing-credential, and model-not-found adapter failures. - Parsed provider reset timestamps with a default one-hour backoff when no usable reset time is present. - Persisted `provider_quota` or `configuration_incomplete` metadata on the failed heartbeat run. - Scheduled quota monitors for the active issue owner, including the current review participant, without creating recovery actions or enqueueing takeover wakes. - Routed configuration failures to blocked recovery with actionable evidence instead of a takeover. - Added unit and embedded-database regression coverage for parsed/fallback quota timing, zero CTO/recovery wake behavior, and configuration classification. ## Verification - `pnpm exec vitest run server/src/services/recovery/provider-failure-classification.test.ts server/src/__tests__/issue-recovery-actions.test.ts server/src/__tests__/issue-monitor-scheduler.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts` — 4 files passed, 66 tests passed. - `pnpm --filter @paperclipai/shared typecheck` — passed. - `pnpm --filter @paperclipai/server typecheck` — passed. ## Risks - Recovery behavior changes for text-matched adapter failures; matching is intentionally conservative, and unmatched failures retain the existing generic recovery path. - Provider reset strings do not always include a date or timezone; parsing chooses the next future matching time and falls back to a one-hour wait when the timestamp is unusable. - This overlaps the provider-quota portion of broader recovery-routing PR #9634, so only one implementation should land if both remain open. - No schema, migration, API contract, or UI changes are included. No documentation update is needed because this corrects internal recovery behavior without changing operator commands or configuration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with model `gpt-5.4`, medium reasoning, tool use, and code execution. The runtime does not expose its configured context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
85404b46c5
commit
c65ab09d9f
8 files changed
+1095
-70
No files matched your search
@@ -1,6 +1,7 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared";
|
||||
import {
|
||||
activityLog,
|
||||
agentRuntimeState,
|
||||
@@ -270,6 +271,73 @@ describeEmbeddedPostgres("issue monitor scheduler", () => {
|
||||
expect(activity).toContain("issue.monitor_triggered");
|
||||
});
|
||||
|
||||
it("wakes a cross-agent review participant for provider quota monitors", async () => {
|
||||
const { companyId, issueId, agentId: assigneeAgentId } = await seedFixture({
|
||||
issueStatus: "in_review",
|
||||
monitor: { serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME },
|
||||
});
|
||||
const participantAgentId = randomUUID();
|
||||
await db.insert(agents).values({
|
||||
id: participantAgentId,
|
||||
companyId,
|
||||
name: "Quota-limited reviewer",
|
||||
role: "engineer",
|
||||
status: "active",
|
||||
adapterType: "process",
|
||||
adapterConfig: {
|
||||
command: process.execPath,
|
||||
args: ["-e", ""],
|
||||
cwd: process.cwd(),
|
||||
},
|
||||
runtimeConfig: {
|
||||
heartbeat: {
|
||||
enabled: false,
|
||||
wakeOnDemand: true,
|
||||
},
|
||||
},
|
||||
permissions: {},
|
||||
});
|
||||
seededAgentIds.add(participantAgentId);
|
||||
const monitorState = await db
|
||||
.select({ executionState: issues.executionState })
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => parseIssueExecutionState(rows[0]?.executionState ?? null)?.monitor ?? null);
|
||||
await db.update(issues).set({
|
||||
executionState: {
|
||||
status: "pending",
|
||||
currentStageId: randomUUID(),
|
||||
currentStageIndex: 0,
|
||||
currentStageType: "review",
|
||||
currentParticipant: { type: "agent", agentId: participantAgentId, userId: null },
|
||||
returnAssignee: { type: "agent", agentId: assigneeAgentId, userId: null },
|
||||
reviewRequest: null,
|
||||
completedStageIds: [],
|
||||
lastDecisionId: null,
|
||||
lastDecisionOutcome: null,
|
||||
monitor: monitorState,
|
||||
},
|
||||
}).where(eq(issues.id, issueId));
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
const result = await heartbeat.tickTimers(new Date("2026-04-11T12:31:00.000Z"));
|
||||
|
||||
expect(result.enqueued).toBe(1);
|
||||
const wakeups = await db.select().from(agentWakeupRequests);
|
||||
expect(wakeups).toHaveLength(1);
|
||||
expect(wakeups[0]).toMatchObject({
|
||||
agentId: participantAgentId,
|
||||
reason: "execution_review_participant_recovery",
|
||||
});
|
||||
await waitForHeartbeatIdle();
|
||||
const participantRuns = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.agentId, participantAgentId));
|
||||
expect(participantRuns).toHaveLength(1);
|
||||
expect(participantRuns[0]?.errorCode).not.toBe("issue_assignee_changed");
|
||||
});
|
||||
|
||||
it("lets the board trigger a scheduled issue monitor immediately", async () => {
|
||||
const { issueId, agentId, nextCheckAt } = await seedFixture();
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
@@ -380,72 +380,538 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("creates a quota wait-recovery monitor without enqueueing a takeover wake", async () => {
|
||||
const { companyId, coderId, sourceIssue } = await seedCompany();
|
||||
it("schedules a provider-quota monitor for the original assignee without creating recovery work", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
const runId = randomUUID();
|
||||
await seedHeartbeatRun({ companyId, agentId: coderId, runId, issueId: sourceIssue.id, status: "failed" });
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "You've hit your usage limit for GPT-5. Try again at 12:00 AM (UTC).",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
const retryAt = new Date(Date.now() + 60 * 60 * 1000).toISOString();
|
||||
|
||||
await recovery.escalateStrandedAssignedIssue({
|
||||
issue: sourceIssue,
|
||||
previousStatus: "in_progress",
|
||||
latestRun: {
|
||||
id: runId,
|
||||
agentId: coderId,
|
||||
status: "failed",
|
||||
error: "provider usage limit exceeded",
|
||||
errorCode: "adapter_failed",
|
||||
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
||||
livenessState: "needs_followup",
|
||||
resultJson: { errorFamily: "provider_quota", providerQuotaRetryNotBefore: retryAt },
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result.providerQuotaMonitored).toBe(1);
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_progress",
|
||||
assigneeAgentId: coderId,
|
||||
monitorScheduledBy: "assignee",
|
||||
monitorNotes: "Provider usage quota reached; retry the original assignee at the provider reset time.",
|
||||
});
|
||||
expect(updatedIssue?.monitorNextCheckAt).toBeInstanceOf(Date);
|
||||
expect(updatedIssue?.executionPolicy).toMatchObject({
|
||||
monitor: {
|
||||
serviceName: "AI provider quota",
|
||||
externalRef: runId,
|
||||
maxAttempts: null,
|
||||
recoveryPolicy: "wake_owner",
|
||||
},
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun).toMatchObject({ errorCode: "provider_quota" });
|
||||
expect(updatedRun?.resultJson).toMatchObject({ errorFamily: "provider_quota" });
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
|
||||
const secondResult = await recovery.reconcileStrandedAssignedIssues();
|
||||
expect(secondResult).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("schedules another provider-quota monitor after a prior quota monitor fired", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
await db.update(issues).set({ monitorAttemptCount: 1 }).where(eq(issues.id, sourceIssueId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T21:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T21:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result.providerQuotaMonitored).toBe(1);
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue?.executionPolicy).toMatchObject({
|
||||
monitor: {
|
||||
maxAttempts: null,
|
||||
externalRef: runId,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("skips provider-quota monitor scheduling for todo issues without aborting reconciliation", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, sourceIssueId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "todo",
|
||||
assigneeAgentId: coderId,
|
||||
monitorNextCheckAt: null,
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not create takeover recovery when a quota monitor cannot be scheduled", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, sourceIssueId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
monitorNextCheckAt: null,
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("schedules a quota monitor for a cross-agent active review participant", async () => {
|
||||
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
||||
const stageId = randomUUID();
|
||||
await db.update(issues).set({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
executionPolicy: {
|
||||
mode: "normal",
|
||||
commentRequired: true,
|
||||
stages: [{
|
||||
id: stageId,
|
||||
type: "review",
|
||||
approvalsNeeded: 1,
|
||||
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
||||
}],
|
||||
},
|
||||
executionState: {
|
||||
status: "pending",
|
||||
currentStageId: stageId,
|
||||
currentStageIndex: 0,
|
||||
currentStageType: "review",
|
||||
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
||||
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
||||
reviewRequest: null,
|
||||
completedStageIds: [],
|
||||
lastDecisionId: null,
|
||||
lastDecisionOutcome: null,
|
||||
},
|
||||
}).where(eq(issues.id, sourceIssueId));
|
||||
const [reviewIssueBeforeRecovery] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(reviewIssueBeforeRecovery).toMatchObject({
|
||||
assigneeAgentId: coderId,
|
||||
executionState: {
|
||||
currentParticipant: { type: "agent", agentId: managerId },
|
||||
returnAssignee: { type: "agent", agentId: coderId },
|
||||
},
|
||||
});
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: managerId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ providerQuotaMonitored: 1, reviewParticipantRequeued: 0 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
monitorNextCheckAt: expect.any(Date),
|
||||
monitorNotes: "Provider usage quota reached; retry the active review participant after the default recovery backoff.",
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("provider_quota");
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not restamp an in_review quota monitor when the assignee has a newer terminal run", async () => {
|
||||
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
||||
const stageId = randomUUID();
|
||||
await db.update(issues).set({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
executionPolicy: {
|
||||
mode: "normal",
|
||||
commentRequired: true,
|
||||
stages: [{
|
||||
id: stageId,
|
||||
type: "review",
|
||||
approvalsNeeded: 1,
|
||||
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
||||
}],
|
||||
},
|
||||
executionState: {
|
||||
status: "pending",
|
||||
currentStageId: stageId,
|
||||
currentStageIndex: 0,
|
||||
currentStageType: "review",
|
||||
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
||||
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
||||
reviewRequest: null,
|
||||
completedStageIds: [],
|
||||
lastDecisionId: null,
|
||||
lastDecisionOutcome: null,
|
||||
},
|
||||
}).where(eq(issues.id, sourceIssueId));
|
||||
const participantRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: participantRunId,
|
||||
companyId,
|
||||
agentId: managerId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const firstResult = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(firstResult).toMatchObject({ providerQuotaMonitored: 1 });
|
||||
const [monitoredIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
const firstNextCheckAt = monitoredIssue?.monitorNextCheckAt;
|
||||
expect(firstNextCheckAt).toBeInstanceOf(Date);
|
||||
expect(monitoredIssue?.executionPolicy).toMatchObject({
|
||||
monitor: {
|
||||
serviceName: "AI provider quota",
|
||||
externalRef: participantRunId,
|
||||
},
|
||||
});
|
||||
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
const [action] = await db
|
||||
.select()
|
||||
.from(issueRecoveryActions)
|
||||
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
||||
expect(action).toMatchObject({
|
||||
cause: "provider_quota",
|
||||
ownerType: "system",
|
||||
ownerAgentId: null,
|
||||
returnOwnerAgentId: coderId,
|
||||
wakePolicy: expect.objectContaining({ type: "monitor_only" }),
|
||||
monitorPolicy: expect.objectContaining({ type: "wait_recovery", retryAgentId: coderId }),
|
||||
});
|
||||
const scheduled = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.status, "scheduled_retry"));
|
||||
expect(scheduled).toHaveLength(1);
|
||||
expect(scheduled[0]).toMatchObject({
|
||||
agentId: coderId,
|
||||
scheduledRetryReason: "provider_quota_recovery",
|
||||
});
|
||||
expect(scheduled[0]?.contextSnapshot).toMatchObject({
|
||||
issueId: sourceIssue.id,
|
||||
wakeReason: "provider_quota_recovery",
|
||||
});
|
||||
expect(scheduled[0]?.contextSnapshot).not.toHaveProperty("recoveryActionId");
|
||||
expect(scheduled[0]?.contextSnapshot).not.toHaveProperty("recoveryCause");
|
||||
const wakePayload = await buildPaperclipWakePayload({
|
||||
db,
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: randomUUID(),
|
||||
companyId,
|
||||
contextSnapshot: scheduled[0]?.contextSnapshot as Record<string, unknown>,
|
||||
agentId: coderId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "Stale assignee wake fired after the issue entered review.",
|
||||
errorCode: "issue_assignee_changed",
|
||||
startedAt: new Date("2026-07-15T20:02:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:03:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
expect(wakePayload?.reason).toBe("provider_quota_recovery");
|
||||
expect(wakePayload?.recovery).toBeNull();
|
||||
const [updatedIssue] = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(eq(issues.id, sourceIssue.id));
|
||||
|
||||
const secondResult = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(secondResult).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
||||
const [unchangedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(unchangedIssue?.monitorNextCheckAt?.getTime()).toBe(firstNextCheckAt?.getTime());
|
||||
expect(unchangedIssue?.executionPolicy).toMatchObject({
|
||||
monitor: {
|
||||
serviceName: "AI provider quota",
|
||||
externalRef: participantRunId,
|
||||
},
|
||||
});
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("classifies review recovery from the active participant run instead of a newer assignee run", async () => {
|
||||
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
||||
const stageId = randomUUID();
|
||||
await db.update(issues).set({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
executionPolicy: {
|
||||
mode: "normal",
|
||||
commentRequired: true,
|
||||
stages: [{
|
||||
id: stageId,
|
||||
type: "review",
|
||||
approvalsNeeded: 1,
|
||||
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
||||
}],
|
||||
},
|
||||
executionState: {
|
||||
status: "pending",
|
||||
currentStageId: stageId,
|
||||
currentStageIndex: 0,
|
||||
currentStageType: "review",
|
||||
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
||||
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
||||
reviewRequest: null,
|
||||
completedStageIds: [],
|
||||
lastDecisionId: null,
|
||||
lastDecisionOutcome: null,
|
||||
},
|
||||
}).where(eq(issues.id, sourceIssueId));
|
||||
const participantRunId = randomUUID();
|
||||
const assigneeRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values([{
|
||||
id: participantRunId,
|
||||
companyId,
|
||||
agentId: managerId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "review process exited unexpectedly",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
}, {
|
||||
id: assigneeRunId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "You've hit your usage limit. Try again at 11:00 PM (UTC)",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:02:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:03:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
}]);
|
||||
const enqueueWakeup = vi.fn(async () => ({ id: randomUUID() } as never));
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ providerQuotaMonitored: 0, reviewParticipantRequeued: 1 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_review",
|
||||
assigneeAgentId: coderId,
|
||||
monitorNextCheckAt: null,
|
||||
});
|
||||
const [assigneeRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, assigneeRunId));
|
||||
expect(assigneeRun?.errorCode).toBe("adapter_failed");
|
||||
expect(enqueueWakeup).toHaveBeenCalledWith(managerId, expect.objectContaining({
|
||||
reason: "execution_review_participant_recovery",
|
||||
payload: expect.objectContaining({ issueId: sourceIssueId, retryOfRunId: participantRunId }),
|
||||
}));
|
||||
});
|
||||
|
||||
it("blocks a cross-agent review participant with incomplete configuration", async () => {
|
||||
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
||||
const stageId = randomUUID();
|
||||
await db.update(issues).set({
|
||||
status: "in_review",
|
||||
executionPolicy: {
|
||||
mode: "normal",
|
||||
commentRequired: true,
|
||||
stages: [{
|
||||
id: stageId,
|
||||
type: "review",
|
||||
approvalsNeeded: 1,
|
||||
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
||||
}],
|
||||
},
|
||||
executionState: {
|
||||
status: "pending",
|
||||
currentStageId: stageId,
|
||||
currentStageIndex: 0,
|
||||
currentStageType: "review",
|
||||
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
||||
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
||||
reviewRequest: null,
|
||||
completedStageIds: [],
|
||||
lastDecisionId: null,
|
||||
lastDecisionOutcome: null,
|
||||
},
|
||||
}).where(eq(issues.id, sourceIssueId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: managerId,
|
||||
invocationSource: "automation",
|
||||
status: "failed",
|
||||
error: "model_not_found: requested review model does not exist",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => ({ id: randomUUID() } as never));
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ escalated: 1, reviewParticipantRequeued: 0 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "blocked",
|
||||
assigneeAgentId: managerId,
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("configuration_incomplete");
|
||||
const [action] = await db.select().from(issueRecoveryActions);
|
||||
expect(action).toMatchObject({
|
||||
sourceIssueId,
|
||||
ownerAgentId: managerId,
|
||||
previousOwnerAgentId: coderId,
|
||||
cause: "configuration_incomplete",
|
||||
recoveryIssueId: null,
|
||||
});
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("uses the default quota backoff when the provider does not state a reset time", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: randomUUID(),
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "Provider quota exceeded for this model.",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result.providerQuotaMonitored).toBe(1);
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_progress",
|
||||
assigneeAgentId: coderId,
|
||||
monitorNotes: "Provider usage quota reached; retry the original assignee after the default recovery backoff.",
|
||||
});
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("classifies model lookup failures as configuration incomplete without waking a recovery owner", async () => {
|
||||
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: coderId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "model_not_found: requested model does not exist",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ escalated: 1, skipped: 0 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue?.status).toBe("blocked");
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("configuration_incomplete");
|
||||
const [action] = await db.select().from(issueRecoveryActions);
|
||||
expect(action).toMatchObject({
|
||||
sourceIssueId,
|
||||
cause: "configuration_incomplete",
|
||||
recoveryIssueId: null,
|
||||
});
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not classify stale configuration failures from a non-assignee run", async () => {
|
||||
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId: managerId,
|
||||
invocationSource: "manual",
|
||||
status: "failed",
|
||||
error: "model_not_found: previous assignee model does not exist",
|
||||
errorCode: "adapter_failed",
|
||||
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
||||
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
||||
contextSnapshot: { issueId: sourceIssueId },
|
||||
});
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
|
||||
const result = await recovery.reconcileStrandedAssignedIssues();
|
||||
|
||||
expect(result).toMatchObject({ escalated: 0, skipped: 1 });
|
||||
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
||||
expect(updatedIssue).toMatchObject({
|
||||
status: "in_progress",
|
||||
assigneeAgentId: coderId,
|
||||
});
|
||||
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
||||
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reuses the same source-scoped action when latest run IDs change while the cause stays the same", async () => {
|
||||
|
||||
Reference in new issue
Block a user