fix: hold an agent's new work while its provider usage limit is active (#19246)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 01:57:31 +01:00 · 2026-11-05 08:57:55 -06:00
1 parent d0afeacdd5
commit e37ab23dbc
@@ -32,7 +32,7 @@ import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
 import * as runnerProbe from "../vendor/paperclip-runner/index.js";
 import * as nativeCodexRunner from "../services/native-runtime/native-codex-runner.js";
 import { testNativeRunnerAuthentication } from "../services/native-runtime/setup-readiness.js";
-import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy, managedAiSessionFingerprintConfig } from "../services/ai-connection-runtime.js";
+import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy, isAiConnectionIncompatible, managedAiSessionFingerprintConfig } from "../services/ai-connection-runtime.js";
 import { execute as executeGemini, testEnvironment as testGeminiEnvironment } from "@paperclipai/adapter-gemini-local/server";
 import { toolAccessService } from "../services/tool-access.js";
 import { secretService } from "../services/secrets.js";
@@ -1949,6 +1949,26 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello"
      await expect(assertManagedAiProjectAuth({ args: ["--api-key=override"] }, "xai", target)).rejects.toThrow("overrides");
    } finally { execute.mockRestore(); }
  });
+  it("reports only a found conflict as configuration, naming the files to fix", async () => {
+    const execute = vi.spyOn(executionTarget, "runAdapterExecutionTargetProcess");
+    const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace/project" } as Parameters<typeof assertManagedAiProjectAuth>[2];
+    const outcome = (exitCode: number | null, timedOut = false) =>
+      ({ exitCode, stdout: "", stderr: "", signal: null, timedOut }) as Awaited<ReturnType<typeof executionTarget.runAdapterExecutionTargetProcess>>;
+    try {
+      execute.mockResolvedValue(outcome(42));
+      const conflict = await assertManagedAiProjectAuth({}, "anthropic", target).catch((error: unknown) => error);
+      expect(isAiConnectionIncompatible(conflict)).toBe(true);
+      expect(String((conflict as Error).message)).toContain(".claude/settings.json");
+      // A check that could not complete is not proof of a configuration conflict.
+      for (const failed of [outcome(43), outcome(null, true), outcome(127)]) {
+        execute.mockResolvedValue(failed);
+        const unavailable = await assertManagedAiProjectAuth({}, "anthropic", target).catch((error: unknown) => error);
+        expect(unavailable).toBeInstanceOf(Error);
+        expect(isAiConnectionIncompatible(unavailable)).toBe(false);
+      }
+      expect(isAiConnectionIncompatible(await assertManagedAiProjectAuth({ args: ["--settings=x"] }, "anthropic", target).catch((error: unknown) => error))).toBe(true);
+    } finally { execute.mockRestore(); }
+  });
  it("requires a typed project authentication capability for task-only environments", async () => {
    const execute = vi.spyOn(executionTarget, "runAdapterExecutionTargetProcess");
    const description = { kind: "description", taskId: "task", projectAuthCheckVersion: 1,
@@ -0,0 +1,197 @@
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db";
+import { registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
+import { heartbeatService } from "../services/heartbeat.js";
+import { createProviderLimitWait } from "../services/heartbeat/provider-limit-wait.js";
+import { executionProjectionsForRuns } from "../services/execution-projection.js";
+import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
+
+const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
+if (!recoveryPostgresSupport.supported) console.warn(`Skipping provider limit wait tests: ${recoveryPostgresSupport.reason}`);
+
+/**
+ * An agent whose provider account is out of allowance must not start new work
+ * that will fail the same way. The wake waits for the reported reset instead.
+ */
+describePostgres("heartbeat provider usage limit wait", () => {
+  let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
+  let db: ReturnType<typeof createDb>;
+  let heartbeat: ReturnType<typeof heartbeatService>;
+  const adapterType = "codex_local";
+  const dispatched = new Set<string>();
+  const oldHome = process.env.PAPERCLIP_HOME;
+  let home: string;
+
+  beforeAll(async () => {
+    temporary = await startRecoveryPostgres("heartbeat-provider-limit-");
+    db = createDb(temporary.connectionString);
+    home = await mkdtemp(path.join(tmpdir(), "paperclip-provider-limit-"));
+    process.env.PAPERCLIP_HOME = home;
+    heartbeat = heartbeatService(db);
+    registerServerAdapter({
+      type: adapterType, supportsLocalAgentJwt: false,
+      execute: async input => {
+        dispatched.add(input.runId);
+        await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId)));
+        return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };
+      },
+      testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }),
+    });
+  }, 60_000);
+
+  afterAll(async () => {
+    await heartbeat?.drainActiveRunExecutions();
+    unregisterServerAdapter(adapterType);
+    await db?.$client.end({ timeout: 0 });
+    await temporary?.cleanup();
+    if (oldHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = oldHome;
+    if (home) await rm(home, { recursive: true, force: true });
+  });
+
+  async function seed(prior: Array<Partial<typeof heartbeatRuns.$inferInsert>>) {
+    const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
+    const userId = `limit-user-${companyId}`;
+    await db.insert(authUsers).values({ id: userId, name: "Limit user", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
+    await db.insert(companies).values({ id: companyId, name: "Provider limit fixture", issuePrefix: `L${companyId.slice(0, 6)}` });
+    await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Limited agent", adapterType, status: "idle", adapterConfig: {} });
+    await db.insert(issues).values({ id: issueId, companyId, title: "Triage the next alert", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1 });
+    for (const values of prior) {
+      await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", triggerDetail: "system", ...values });
+    }
+    return { companyId, agentId, issueId };
+  }
+
+  async function wake(f: Awaited<ReturnType<typeof seed>>, manual = false) {
+    const run = await heartbeat.wakeup(f.agentId, manual
+      ? { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } }
+      : { source: "automation", triggerDetail: "system", reason: "issue_assigned", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
+    expect(run).not.toBeNull();
+    await heartbeat.drainActiveRunExecutions();
+    const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
+    return finished;
+  }
+
+  const rateLimited = (minutesAgo: number, overrides: Record<string, unknown> = {}) => ({
+    status: "failed",
+    startedAt: new Date(Date.now() - (minutesAgo + 2) * 60_000),
+    finishedAt: new Date(Date.now() - minutesAgo * 60_000),
+    errorCode: "provider_session_open_rejected",
+    resultJson: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
+      category: "rate_limit", httpStatus: 429, retryability: "unknown",
+      message: "API error: 429 This request would exceed your account's rate limit. Please try again later", ...overrides } },
+  });
+
+  it("defers new work until the reported reset instead of dispatching a doomed run", async () => {
+    const limited = rateLimited(16);
+    const f = await seed([limited]);
+    const resetAt = new Date(limited.finishedAt.getTime() + 60 * 60_000);
+    const finished = await wake(f);
+
+    expect(dispatched.has(finished.id)).toBe(false);
+    expect(finished).toMatchObject({
+      status: "cancelled",
+      errorCode: "provider_limit_wait",
+      resultJson: {
+        executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false },
+        providerLimitWait: { resetAt: resetAt.toISOString(), resetReported: false, category: "rate_limit" },
+      },
+    });
+    expect(finished.error).toContain(resetAt.toISOString());
+    const successors = await db.select().from(heartbeatRuns)
+      .where(eq(heartbeatRuns.agentId, f.agentId));
+    const retry = successors.find(run => run.status === "scheduled_retry");
+    expect(retry).toMatchObject({ scheduledRetryReason: "ai_connection_pool_wait" });
+    expect(retry!.scheduledRetryAt!.getTime()).toBeGreaterThanOrEqual(resetAt.getTime());
+    expect(retry!.scheduledRetryAt!.getTime()).toBeLessThan(resetAt.getTime() + 5 * 60_000);
+    const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id));
+    expect(events.some(event => event.message?.includes("provider usage limit"))).toBe(true);
+    const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    expect(issue.status).not.toBe("blocked");
+    // The persisted projection names the wait and its reset.
+    expect((await executionProjectionsForRuns(db, f.companyId, [retry!.id])).get(retry!.id)).toMatchObject({
+      label: "Waiting for usage limit reset", phase: "retry_scheduled",
+      nextAction: expect.stringContaining(retry!.scheduledRetryAt!.toISOString()),
+    });
+
+    // Nothing is dropped: the wait starts the work once the reset has passed.
+    expect((await heartbeat.promoteDueScheduledRetries(new Date())).runIds).not.toContain(retry!.id);
+    // Model the hour passing by moving the limit evidence back past its reset.
+    await db.update(heartbeatRuns).set({ startedAt: new Date(Date.now() - 3 * 60 * 60_000), finishedAt: new Date(Date.now() - 2 * 60 * 60_000) })
+      .where(eq(heartbeatRuns.errorCode, "provider_session_open_rejected"));
+    expect((await heartbeat.promoteDueScheduledRetries(new Date(retry!.scheduledRetryAt!.getTime() + 1))).runIds).toContain(retry!.id);
+    await heartbeat.resumeQueuedRuns();
+    await heartbeat.drainActiveRunExecutions();
+    expect(dispatched.has(retry!.id)).toBe(true);
+    const [resumed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retry!.id));
+    expect(resumed.status).toBe("succeeded");
+  }, 60_000);
+
+  it("releases the task when its retry cannot be scheduled, even if diagnostics fail", async () => {
+    const f = await seed([rateLimited(16)]);
+    const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
+    const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
+      invocationSource: "automation", triggerDetail: "system", contextSnapshot: { issueId: f.issueId } }).returning();
+    const deps = {
+      setRunStatusIfRunning: vi.fn(async () => ({ updated: true, run: { ...run, status: "cancelled" } })),
+      setWakeupStatus: vi.fn(async () => { throw new Error("wakeup write failed"); }),
+      appendRunEvent: vi.fn(async () => { throw new Error("event write failed"); }),
+      scheduleBoundedRetryForRun: vi.fn(async () => { throw new Error("retry write failed"); }),
+      releaseIssueExecutionAndPromote: vi.fn(async () => undefined),
+      finalizeAgentStatus: vi.fn(async () => undefined),
+    };
+    const { deferRunForProviderLimit } = createProviderLimitWait(db, deps as never);
+    await expect(deferRunForProviderLimit(run, agent, { resumingAdmittedRun: false, conversation: false })).resolves.toBe(true);
+    expect(deps.scheduleBoundedRetryForRun).toHaveBeenCalledTimes(2);
+    expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledOnce();
+    expect(deps.finalizeAgentStatus).toHaveBeenCalledOnce();
+
+    // A scheduled retry keeps the task lock for its successor.
+    deps.scheduleBoundedRetryForRun.mockReset().mockResolvedValue({ outcome: "scheduled" } as never);
+    deps.releaseIssueExecutionAndPromote.mockClear();
+    await expect(deferRunForProviderLimit(run, agent, { resumingAdmittedRun: false, conversation: false })).resolves.toBe(true);
+    expect(deps.releaseIssueExecutionAndPromote).not.toHaveBeenCalled();
+  });
+
+  it("recovers a wait whose retry was lost to a crash after its cancellation committed", async () => {
+    const f = await seed([]);
+    const [lost] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "cancelled",
+      invocationSource: "automation", triggerDetail: "system", issueId: f.issueId, scopeKind: "issue",
+      startedAt: new Date(Date.now() - 10 * 60_000), finishedAt: new Date(Date.now() - 9 * 60_000), errorCode: "provider_limit_wait",
+      resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } },
+      contextSnapshot: { issueId: f.issueId } }).returning();
+    await db.update(issues).set({ status: "in_progress", executionRunId: lost.id, executionLockedAt: new Date(Date.now() - 9 * 60_000) })
+      .where(eq(issues.id, f.issueId));
+    await heartbeat.sweepStaleIssueLocks();
+    await heartbeat.reconcileStrandedAssignedIssues();
+    await heartbeat.drainActiveRunExecutions();
+    const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agentId));
+    // Existing stranded-work recovery gives the wake a bounded successor; the gate re-checks the limit when it runs.
+    expect(runs.find(run => run.id !== lost.id)).toMatchObject({ status: "scheduled_retry", retryOfRunId: lost.id });
+    const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    expect(issue).toMatchObject({ status: "in_progress", executionRunId: null });
+  }, 60_000);
+
+  it("still dispatches a manual run, and work after a later success or an expired reset", async () => {
+    const manual = await seed([rateLimited(16)]);
+    expect(dispatched.has((await wake(manual, true)).id)).toBe(true);
+
+    const recovered = await seed([rateLimited(30), {
+      status: "succeeded", startedAt: new Date(Date.now() - 10 * 60_000), finishedAt: new Date(Date.now() - 9 * 60_000),
+    }]);
+    expect(dispatched.has((await wake(recovered)).id)).toBe(true);
+
+    const expired = await seed([rateLimited(90)]);
+    expect(dispatched.has((await wake(expired)).id)).toBe(true);
+
+    const unrelated = await seed([{ ...rateLimited(5), resultJson: { providerFailure: {
+      schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "turn", category: "unknown",
+      retryability: "unknown", message: "ACP agent reported a terminal limit failure." } } }]);
+    expect(dispatched.has((await wake(unrelated)).id)).toBe(true);
+  }, 60_000);
+});
@@ -34,6 +34,19 @@ export function isAiConnectionBusy(error: unknown): error is HttpError {
    (error.details as { code?: unknown } | undefined)?.code === "ai_connection_busy";
 }

+/** A real conflict between the agent's AI connection and its project or arguments. */
+export function isAiConnectionIncompatible(error: unknown): error is HttpError {
+  return error instanceof HttpError && error.status === 422 &&
+    (error.details as { code?: unknown } | undefined)?.code === "ai_connection_incompatible";
+}
+
+function projectAuthConflict(files: string[]) {
+  return unprocessable(
+    `This agent's managed AI connection conflicts with project authentication settings in ${files.join(" or ")}. Remove the API key, token or apiKeyHelper settings from that project file, or choose a different AI connection for this agent.`,
+    { code: "ai_connection_incompatible" },
+  );
+}
+
 // Blank values intentionally override inherited credentials in CLI child environments.
 export const AI_AUTH_ENV_KEYS = [
  "COPILOT_GITHUB_TOKEN",
@@ -162,10 +175,11 @@ done`,
        onLog: async () => {},
      },
    );
+    if (result.exitCode === 42) throw projectAuthConflict(files);
+    // The check fails closed, but an unfinished check is not a found conflict.
    if (result.exitCode !== 0)
-      throw unprocessable(
-        "The environment's project authentication settings must be checked before using this AI connection",
-        { code: "ai_connection_incompatible" },
+      throw new Error(
+        `Paperclip could not check the environment's project authentication settings (${result.timedOut ? "timed out" : `exit ${result.exitCode ?? result.signal ?? "unknown"}`}). The run did not start; it can be retried.`,
      );
    return;
  }
@@ -180,10 +194,7 @@ done`,
            content,
          )
        ) {
-          throw unprocessable(
-            "Project authentication settings conflict with the selected AI connection",
-            { code: "ai_connection_incompatible" },
-          );
+          throw projectAuthConflict([relative]);
        }
      } catch (error) {
        if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
@@ -101,6 +101,15 @@ describe("execution truth projection", () => {
      scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeAiConnectionWait: 0 } }), undefined, [], undefined, now))
      .toMatchObject({ label: "Waiting for AI subscription", phase: "retry_scheduled", attempt: 1, recoveryOwner: null });
  });
+  it("shows a provider usage limit wait with its reset time", () => {
+    const scheduledRetryAt = new Date("2026-11-05T13:47:50.375Z");
+    expect(projectExecution(run({ runtimeMode: "native", status: "scheduled_retry", scheduledRetryReason: "ai_connection_pool_wait",
+      scheduledRetryAt, scheduledRetryAttempt: 3, contextSnapshot: { wakeReason: "provider_limit_retry", failureRetriesBeforeAiConnectionWait: 0 } }),
+    undefined, [], undefined, now)).toMatchObject({
+      label: "Waiting for usage limit reset", phase: "retry_scheduled", attempt: 1, recoveryOwner: null,
+      nextAction: expect.stringContaining(scheduledRetryAt.toISOString()),
+    });
+  });
  it("shows a workspace wait without presenting its deferral count as failed attempts", () => {
    expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "workspace_busy",
      scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } }), undefined, [], undefined, now))
@@ -37,6 +37,7 @@ const executionRunColumns = {
  status: heartbeatRuns.status,
  contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
    'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
+    'wakeReason', ${heartbeatRuns.contextSnapshot}->'wakeReason',
    'failureRetriesBeforeAiConnectionWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeAiConnectionWait',
    'failureRetriesBeforeWorkspaceWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeWorkspaceWait')`,
 };
@@ -302,6 +303,12 @@ export function projectExecution(
    projection.nextAction = "Waiting for the AI subscription's current execution to finish; the scheduled check will revalidate access.";
    return set("retry_scheduled", "Waiting for AI subscription");
  }
+  if (run.status === "scheduled_retry" && run.scheduledRetryReason === "ai_connection_pool_wait"
+      && run.contextSnapshot?.wakeReason === "provider_limit_retry") {
+    const resetAt = run.scheduledRetryAt ? new Date(run.scheduledRetryAt).toISOString() : "its reported reset";
+    projection.nextAction = `Waiting for the agent's provider usage limit to reset; the work starts at ${resetAt}. To continue sooner, restore the provider allowance or change the agent's AI connection, then start a manual run.`;
+    return set("retry_scheduled", "Waiting for usage limit reset");
+  }
  if (run.status === "scheduled_retry" && run.scheduledRetryReason === "workspace_busy") {
    projection.nextAction = "Waiting for the live workspace holder to finish; the scheduled check will revalidate ownership.";
    return set("retry_scheduled", "Waiting for workspace");
@@ -5,6 +5,7 @@ import { registerIssueRecoveryWork } from "./recovery/work.js";
 import { registerRunDispatchWork } from "./heartbeat/dispatch-work.js";
 import { NATIVE_RETAINED_OWNER_WAIT } from "./native-runtime/native-controller-recovery.js";
 import { describeFailedAttachment } from "./native-runtime/remote-warm-session-recovery.js";
+import { createProviderLimitWait } from "./heartbeat/provider-limit-wait.js";
 import { externalConversationSettlementWork } from "./external-conversation-settlement-work.js";
 import { nativeRecoveryNoticeWork } from "./native-runtime/native-recovery-notice-work.js";
 import { nativeRunProjectionWork } from "./native-runtime/native-run-projection-work.js";
@@ -288,6 +289,7 @@ import { retryChatControlAdmission } from "./chat-control-admission-retry.js";
 import {
  prepareManagedAiRuntime,
  assertManagedAiProjectAuth,
+  isAiConnectionIncompatible,
 } from "./ai-connection-runtime.js";
 import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared";
 import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js";
@@ -1837,6 +1839,15 @@ export function heartbeatService(
    return withRunExecutionTiming(runId, () => executeRunAttempt(runId, runOptions));
  }

+  const { deferRunForProviderLimit } = createProviderLimitWait(db, {
+    setRunStatusIfRunning,
+    setWakeupStatus,
+    appendRunEvent,
+    scheduleBoundedRetryForRun,
+    releaseIssueExecutionAndPromote,
+    finalizeAgentStatus,
+  });
+
  async function executeRunAttempt(
    runId: string,
    runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> },
@@ -2069,6 +2080,10 @@ export function heartbeatService(
        await finalizeAgentStatus(agent.id, "cancelled");
        return;
      }
+      if (await deferRunForProviderLimit(run, agent, {
+        resumingAdmittedRun: !!runOptions.nativeLeaseOwner,
+        conversation: isConversation(conversationIdentity),
+      })) return;
      run = await timePreparation("chat_completion", () => prepareChatCompletionTurn(db, run));
      const preparedConversation = await timePreparation("conversation", () => prepareConversationTurn(db, run));
      run = { ...run, contextSnapshot: preparedConversation.context };
@@ -2810,7 +2825,11 @@ export function heartbeatService(
      });
      if (managedAiRuntime && aiBinding) {
        try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); }
-        catch { throw new ConfigurationIncompleteFailure("Project authentication conflicts with this agent’s managed AI connection", { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } }); }
+        catch (error) {
+          // Only a found conflict is the board's to fix; an unfinished check stays an ordinary failure.
+          if (!isAiConnectionIncompatible(error)) throw error;
+          throw new ConfigurationIncompleteFailure(error.message, { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } });
+        }
      }
      const remoteExecution = realizationResult.remoteExecution;
      if (nativeChatWorkspaceScope && materializeNativeChatWorktree) {
@@ -0,0 +1,117 @@
+import type { Db } from "@paperclipai/db";
+import { agents, heartbeatRuns } from "@paperclipai/db";
+import { logger } from "../../middleware/logger.js";
+import { AI_CONNECTION_POOL_WAIT_RETRY_REASON } from "../../modules/run-dispatch/index.js";
+import { agentProviderLimitCooldown } from "../native-runtime/native-startup-retry.js";
+import { parseObject } from "../../adapters/utils.js";
+
+type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
+type Agent = typeof agents.$inferSelect;
+
+export interface ProviderLimitWaitDependencies {
+  setRunStatusIfRunning: (runId: string, status: "cancelled", patch: {
+    error: string; errorCode: string; finishedAt: Date; resultJson: Record<string, unknown>;
+  }) => Promise<{ updated: boolean; run?: HeartbeatRun | null }>;
+  setWakeupStatus: (wakeupRequestId: string | null | undefined, status: "cancelled", patch: { finishedAt: Date; error: string }) => Promise<unknown>;
+  appendRunEvent: (run: HeartbeatRun, event: {
+    eventType: string; stream: "system"; level: "warn"; message: string; payload: Record<string, unknown>;
+  }) => Promise<unknown>;
+  scheduleBoundedRetryForRun: (run: HeartbeatRun, agent: Agent, opts: {
+    now: Date; retryReason: string; wakeReason: string; maxAttempts: number; delayMs: number;
+  }) => Promise<{ outcome: string }>;
+  releaseIssueExecutionAndPromote: (run: HeartbeatRun) => Promise<unknown>;
+  finalizeAgentStatus: (agentId: string, outcome: "cancelled") => Promise<unknown>;
+}
+
+/** Holds new work while the agent's provider account is out of allowance. */
+export function createProviderLimitWait(db: Db, deps: ProviderLimitWaitDependencies) {
+  const {
+    setRunStatusIfRunning, setWakeupStatus, appendRunEvent, scheduleBoundedRetryForRun,
+    releaseIssueExecutionAndPromote, finalizeAgentStatus,
+  } = deps;
+
+  /**
+   * An agent whose provider account recently reported an exhausted allowance
+   * would fail new work the same way. Before any workspace or provider work,
+   * hold the wake until the reported reset and retry it then. A manual run, a
+   * recovering run, a conversation turn, and pool routing (which selects
+   * another account) still start, so an operator can always test a repair.
+   */
+  async function deferRunForProviderLimit(
+    run: HeartbeatRun,
+    agent: Agent,
+    options: { resumingAdmittedRun: boolean; conversation: boolean },
+  ): Promise<boolean> {
+    if (options.resumingAdmittedRun || options.conversation) return false;
+    if (run.invocationSource === "on_demand" && run.triggerDetail === "manual") return false;
+    if (parseObject(parseObject(agent.runtimeConfig).aiConnection).mode === "router") return false;
+    const now = new Date();
+    const cooldown = await agentProviderLimitCooldown(db, {
+      companyId: run.companyId, agentId: run.agentId, runId: run.id, now,
+    }).catch((err: unknown) => {
+      // Missing evidence must not stop work; the provider still reports its own limit.
+      logger.warn({ err, runId: run.id }, "provider limit evidence unavailable before dispatch");
+      return null;
+    });
+    if (!cooldown) return false;
+    const resetAt = cooldown.resetAt.toISOString();
+    const message = `The provider usage limit for this agent is expected to reset at ${resetAt}${
+      cooldown.resetReported ? "" : " (no reset time was reported, so Paperclip waits one hour)"
+    }. Paperclip will start this work then.`;
+    const cancelled = await setRunStatusIfRunning(run.id, "cancelled", {
+      error: message, errorCode: "provider_limit_wait", finishedAt: now,
+      resultJson: {
+        executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false },
+        providerLimitWait: {
+          resetAt, resetReported: cooldown.resetReported, category: cooldown.category,
+          sourceRunId: cooldown.sourceRunId, actionUrl: `/agents/${agent.id}/runtime`,
+        },
+        cancellation: {
+          source: "control_plane", expected: true, initiator: { type: "system" },
+          reason: "Waiting for the agent's provider usage limit to reset", recordedAt: now.toISOString(),
+        },
+        retryAt: resetAt,
+      },
+    });
+    if (!cancelled.updated) return true;
+    const deferred = cancelled.run ?? run;
+    // The cancellation is committed. From here the wake must either gain its
+    // retry or release its task lock; diagnostics never decide that.
+    let scheduled = false;
+    try {
+      // Scheduling is idempotent per source run, so one more attempt is safe.
+      for (let attempt = 1; attempt <= 2 && !scheduled; attempt += 1) {
+        try {
+          // Each wait is one retry at the reset; evidence expires within a day.
+          const retry = await scheduleBoundedRetryForRun(deferred, agent, {
+            now, retryReason: AI_CONNECTION_POOL_WAIT_RETRY_REASON, wakeReason: "provider_limit_retry",
+            maxAttempts: (run.scheduledRetryAttempt ?? 0) + 1,
+            delayMs: Math.max(1000, cooldown.resetAt.getTime() - now.getTime() + 1000),
+          });
+          scheduled = retry.outcome === "scheduled";
+          if (!scheduled) break;
+        } catch (err) {
+          logger.warn({ err, runId: run.id, attempt }, "failed to schedule the provider limit retry");
+        }
+      }
+      await appendRunEvent(deferred, {
+        eventType: "lifecycle", stream: "system", level: "warn",
+        message: scheduled
+          ? `Waiting for the provider usage limit to reset at ${resetAt} before starting work`
+          : `The provider usage limit resets at ${resetAt}; this wake could not schedule its retry and released the task`,
+        payload: { resetAt, resetReported: cooldown.resetReported, category: cooldown.category, sourceRunId: cooldown.sourceRunId, retryScheduled: scheduled },
+      }).catch((err: unknown) => logger.warn({ err, runId: run.id }, "failed to record the provider limit wait"));
+      await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: now, error: message })
+        .catch((err: unknown) => logger.warn({ err, runId: run.id }, "failed to settle the provider limit wakeup"));
+    } finally {
+      try {
+        if (!scheduled) await releaseIssueExecutionAndPromote(deferred);
+      } finally {
+        await finalizeAgentStatus(run.agentId, "cancelled");
+      }
+    }
+    return true;
+  }
+
+  return { deferRunForProviderLimit };
+}
@@ -69,7 +69,7 @@ export function legacyExecutionNeedsReconciliation(
      && (run.resultJson?.executionCancellation as Record<string, unknown> | undefined)?.state === "acknowledged") return false;
  // Waiting for a subscription or workspace precedes provider execution. It is
  // a resource wait, not a failed provider attempt or permission to replay work.
-  if (run.status === "cancelled" && (run.errorCode === "ai_connection_busy" || run.errorCode === "ai_connection_pool_exhausted") &&
+  if (run.status === "cancelled" && (run.errorCode === "ai_connection_busy" || run.errorCode === "ai_connection_pool_exhausted" || run.errorCode === "provider_limit_wait") &&
      evidence?.kind === "ai_connection_wait" && evidence.providerWorkStarted === false) return false;
  if (run.status === "cancelled" && run.errorCode === "workspace_busy" &&
      evidence?.kind === "workspace_wait" && evidence.providerWorkStarted === false) return false;
@@ -12842,6 +12842,30 @@ describe("native session bounded recovery", () => {
        payload: { status: "failed", error: { message: "usageLimitExceeded" } },
      }),
    ).toBe(false);
+    // ACP harnesses report the same terminal through a typed provider failure.
+    const acpLimit = (category: string) => ({
+      ...event,
+      payload: {
+        status: "failed",
+        error: {
+          code: "RUNTIME",
+          message: "ACP agent reported a terminal limit failure.",
+          providerFailure: {
+            schema: "paperclip.provider-failure.v1",
+            provider: "anthropic",
+            phase: "turn",
+            category,
+            message: "ACP agent reported a terminal limit failure.\nYou've hit your session limit · resets 2am (UTC)",
+            retryability: "unknown",
+          },
+        },
+      },
+    });
+    expect(nativeProviderUsageLimitFromEvent(acpLimit("quota_exhausted"))).toBe(true);
+    expect(nativeProviderUsageLimitFromEvent(acpLimit("rate_limit"))).toBe(true);
+    // A context, turn or budget limit is not an account allowance.
+    expect(nativeProviderUsageLimitFromEvent(acpLimit("unknown"))).toBe(false);
+    expect(nativeProviderUsageLimitFromEvent({ ...acpLimit("quota_exhausted"), sourceKind: "provider" as never })).toBe(false);
    expect(
      nativeSessionFailureDisposition(
        1,
@@ -6,6 +6,7 @@ import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES, nativeHarnessBackupManifestSha256,
 import { prepareCursorCloudRuntime } from "./cursor-cloud-runtime.js";
 import { resolveLocalRuntimeApiUrl } from "../../runtime-api.js";
 import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js";
+import { readProviderFailure } from "@paperclipai/shared";
 import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js";
 import { readClaimedNativeStopRecovery } from "./native-cancellation-request.js";
 import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js";
@@ -7123,12 +7124,13 @@ export function nativeProviderUsageLimitFromEvent(
  event: Pick<PrpEvent, "sourceKind" | "eventType" | "payload">,
 ): boolean {
  const payload = record(event.payload);
-  return (
-    event.sourceKind === "runner" &&
-    event.eventType === "turn.failed" &&
-    payload.status === "failed" &&
-    record(payload.error).codexErrorInfo === "usageLimitExceeded"
-  );
+  if (event.sourceKind !== "runner" || event.eventType !== "turn.failed" || payload.status !== "failed") return false;
+  const error = record(payload.error);
+  if (error.codexErrorInfo === "usageLimitExceeded") return true;
+  // ACP harnesses type the same exhausted allowance as a provider failure.
+  // Context, turn and configured budget limits stay ordinary terminals.
+  const category = readProviderFailure(error.providerFailure)?.category;
+  return category === "quota_exhausted" || category === "rate_limit";
 }

 export function nativeSessionFailureDisposition(
@@ -39,6 +39,7 @@ import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
 import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk";
 import { accountRunCost } from "../run-cost-accounting.js";
 import { settleUnrecoverableExecutions } from "../execution-recovery-resolution.js";
+import { agentProviderLimitCooldown } from "./native-startup-retry.js";

 type SessionOpenStatus = "failed" | "rejected" | "indeterminate";

@@ -659,6 +660,34 @@ describe("native provider startup retry", () => {
    expect(state.issue.status).toBe("in_progress");
  });

+  it("reads the reset clock from an ACP terminal limit failure's provider wording", async () => {
+    const agentId = await createAgent();
+    const limitedAt = new Date(Date.now() - 5 * 60_000);
+    const resetHour = new Date(Date.now() + 3 * 60 * 60_000).getUTCHours();
+    await recordPriorRun(agentId, {
+      status: "failed",
+      startedAt: new Date(limitedAt.getTime() - 60_000),
+      finishedAt: limitedAt,
+      errorCode: "native_provider_usage_limit",
+      resultJson: {
+        providerFailure: limitFailure({
+          category: "quota_exhausted",
+          message: `ACP agent reported a terminal limit failure.\nYou've hit your session limit · resets ${resetHour}:00 (UTC)`,
+        }),
+      },
+    });
+    const expected = new Date(limitedAt);
+    expected.setUTCHours(resetHour, 0, 0, 0);
+    if (expected <= limitedAt) expected.setUTCDate(expected.getUTCDate() + 1);
+
+    const seeded = await seedRun(agentId);
+    await failStartup(seeded, sessionOpenFailure());
+    const state = await read(seeded);
+    expect(state.coordinator.failureDetail).toMatchObject({ providerCooldownAt: expected.toISOString() });
+    expect(await agentProviderLimitCooldown(db, { companyId, agentId, now: new Date() }))
+      .toMatchObject({ resetAt: expected, resetReported: true });
+  });
+
  it("holds a startup retry for at most one day before trying again", async () => {
    const seeded = await seedRun(await createAgent());
    const failedAt = await failStartup(seeded, sessionOpenFailure({
@@ -4,6 +4,7 @@ import { heartbeatRuns, type Db } from "@paperclipai/db";
 import { readProviderFailure, type ProviderFailure } from "@paperclipai/shared";
 import { logger } from "../../middleware/logger.js";
 import { infrastructureRetryDelayMs, PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
+import { parseProviderQuotaClockReset } from "../provider-quota-reset.js";

 /** Categories that report an exhausted account or request allowance. */
 const PROVIDER_LIMIT_CATEGORIES = ["rate_limit", "quota_exhausted"];
@@ -13,26 +14,46 @@ const PROVIDER_LIMIT_ERROR_CODES = ["native_provider_usage_limit", "provider_quo
 const NATIVE_PROVIDER_LIMIT_WINDOW_MS = 24 * 60 * 60_000;
 const LIMIT_EVIDENCE_ROWS = 20;

-function limitCooldownAt(failure: ProviderFailure | null, observedAt: Date): number | null {
+type ProviderLimitEvidence = { resetAt: number; resetReported: boolean; category: string; sourceRunId: string | null };
+
+function limitEvidence(failure: ProviderFailure | null, observedAt: Date, sourceRunId: string | null): ProviderLimitEvidence | null {
  if (!failure || !PROVIDER_LIMIT_CATEGORIES.includes(failure.category)) return null;
  const retryAt = failure.retryAt ? Date.parse(failure.retryAt) : Number.NaN;
-  return Number.isFinite(retryAt) ? retryAt : observedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS;
+  if (Number.isFinite(retryAt)) return { resetAt: retryAt, resetReported: true, category: failure.category, sourceRunId };
+  // ACP harnesses report the reset only in the provider's own wording.
+  const clockReset = parseProviderQuotaClockReset(failure.message, observedAt);
+  return clockReset
+    ? { resetAt: clockReset.getTime(), resetReported: true, category: failure.category, sourceRunId }
+    : { resetAt: observedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS, resetReported: false, category: failure.category, sourceRunId };
 }

+export type AgentProviderLimitCooldown = {
+  /** When the agent's provider allowance is expected to return. */
+  resetAt: Date;
+  /** False when no reset was reported and the default cooldown applies. */
+  resetReported: boolean;
+  category: string;
+  /** The run whose failure recorded the latest reset. */
+  sourceRunId: string | null;
+};
+
 /**
 * The latest known provider limit for this agent, from the current failure or
 * from the agent's recent runs. A later successful run clears older evidence.
 */
-async function nativeProviderCooldownAt(input: {
-  db: Db;
-  binding: { companyId: string; agentId: string; runId: string };
-  failure: ProviderFailure | null;
+export async function agentProviderLimitCooldown(db: Db, input: {
+  companyId: string;
+  agentId: string;
+  /** Excluded from the evidence; the caller supplies its own failure. */
+  runId?: string;
+  failure?: ProviderFailure | null;
  now: Date;
-}): Promise<Date | null> {
+}): Promise<AgentProviderLimitCooldown | null> {
  const now = input.now.getTime();
-  let cooldownAt = limitCooldownAt(input.failure, input.now) ?? 0;
-  const rows = await input.db
+  let latest = limitEvidence(input.failure ?? null, input.now, input.runId ?? null);
+  const rows = await db
    .select({
+      id: heartbeatRuns.id,
      status: heartbeatRuns.status,
      errorCode: heartbeatRuns.errorCode,
      observedAt: sql<Date>`coalesce(${heartbeatRuns.finishedAt}, ${heartbeatRuns.updatedAt})`.mapWith(heartbeatRuns.updatedAt),
@@ -41,10 +62,10 @@ async function nativeProviderCooldownAt(input: {
    })
    .from(heartbeatRuns)
    .where(and(
-      eq(heartbeatRuns.companyId, input.binding.companyId),
-      eq(heartbeatRuns.agentId, input.binding.agentId),
+      eq(heartbeatRuns.companyId, input.companyId),
+      eq(heartbeatRuns.agentId, input.agentId),
      gte(heartbeatRuns.startedAt, new Date(now - NATIVE_PROVIDER_LIMIT_WINDOW_MS)),
-      ne(heartbeatRuns.id, input.binding.runId),
+      ...(input.runId ? [ne(heartbeatRuns.id, input.runId)] : []),
      or(
        eq(heartbeatRuns.status, "succeeded"),
        inArray(heartbeatRuns.errorCode, PROVIDER_LIMIT_ERROR_CODES),
@@ -56,15 +77,22 @@ async function nativeProviderCooldownAt(input: {
  for (const row of rows) {
    // The provider accepted work after this point, so its allowance was restored.
    if (row.status === "succeeded") break;
-    const failure = readProviderFailure(row.providerFailure);
-    let recorded = limitCooldownAt(failure, row.observedAt);
+    let recorded = limitEvidence(readProviderFailure(row.providerFailure), row.observedAt, row.id);
    if (recorded === null && PROVIDER_LIMIT_ERROR_CODES.includes(row.errorCode ?? "")) {
      const reset = row.retryNotBefore ? Date.parse(row.retryNotBefore) : Number.NaN;
-      recorded = Number.isFinite(reset) ? reset : row.observedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS;
+      recorded = Number.isFinite(reset)
+        ? { resetAt: reset, resetReported: true, category: "quota_exhausted", sourceRunId: row.id }
+        : { resetAt: row.observedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS, resetReported: false, category: "quota_exhausted", sourceRunId: row.id };
    }
-    if (recorded !== null) cooldownAt = Math.max(cooldownAt, recorded);
+    if (recorded && (!latest || recorded.resetAt > latest.resetAt)) latest = recorded;
  }
-  return cooldownAt > now ? new Date(Math.min(cooldownAt, now + NATIVE_PROVIDER_LIMIT_WINDOW_MS)) : null;
+  if (!latest || latest.resetAt <= now) return null;
+  return {
+    resetAt: new Date(Math.min(latest.resetAt, now + NATIVE_PROVIDER_LIMIT_WINDOW_MS)),
+    resetReported: latest.resetReported,
+    category: latest.category,
+    sourceRunId: latest.sourceRunId,
+  };
 }

 /**
@@ -82,7 +110,13 @@ export async function planNativeStartupRetry(input: {
 }): Promise<{ nextAttemptAt: Date | null; providerCooldownAt: Date | null }> {
  // Limit evidence only refines the schedule. A failed read must not stop the
  // failure from settling, so the retry falls back to plain backoff.
-  const providerCooldownAt = await nativeProviderCooldownAt(input).catch((error: unknown) => {
+  const providerCooldownAt = await agentProviderLimitCooldown(input.db, {
+    companyId: input.binding.companyId,
+    agentId: input.binding.agentId,
+    runId: input.binding.runId,
+    failure: input.failure,
+    now: input.now,
+  }).then((cooldown) => cooldown?.resetAt ?? null, (error: unknown) => {
    logger.warn({ err: error, runId: input.binding.runId }, "native provider limit evidence unavailable");
    return null;
  });
@@ -0,0 +1,81 @@
+/**
+ * Reads a provider's "try again at 4:30 PM (America/Chicago)" or
+ * "resets 2am (UTC)" wording as the next matching wall-clock instant after
+ * `now`. A clock without a time zone is read as UTC.
+ */
+export function parseProviderQuotaClockReset(error: string, now: Date) {
+  const match = error.match(
+    /(?:try again at|resets?(?:\s+at)?)\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i,
+  );
+  if (!match) return null;
+
+  const hourValue = Number.parseInt(match[1] ?? "", 10);
+  const minute = Number.parseInt(match[2] ?? "0", 10);
+  const meridiem = (match[3] ?? "").toLowerCase();
+  if (!Number.isInteger(hourValue)) return null;
+  if (
+    meridiem ? hourValue < 1 || hourValue > 12 : hourValue < 0 || hourValue > 23
+  )
+    return null;
+  if (!Number.isInteger(minute) || minute < 0 || minute > 59) return null;
+
+  let hour = meridiem ? hourValue % 12 : hourValue;
+  if (meridiem === "p") hour += 12;
+  const timeZone = (match[4] ?? match[5])?.trim();
+  if (!timeZone) {
+    const retryAt = new Date(now);
+    retryAt.setUTCHours(hour, minute, 0, 0);
+    if (retryAt.getTime() <= now.getTime())
+      retryAt.setUTCDate(retryAt.getUTCDate() + 1);
+    return retryAt;
+  }
+
+  try {
+    const wallClock = (date: Date) =>
+      Object.fromEntries(
+        new Intl.DateTimeFormat("en-US", {
+          timeZone,
+          hourCycle: "h23",
+          year: "numeric",
+          month: "2-digit",
+          day: "2-digit",
+          hour: "2-digit",
+          minute: "2-digit",
+        })
+          .formatToParts(date)
+          .map((part) => [part.type, part.value]),
+      );
+    const nowParts = wallClock(now);
+    const buildRetryAt = (dayOffset: number) => {
+      const targetDay = new Date(
+        Date.UTC(
+          Number(nowParts.year),
+          Number(nowParts.month) - 1,
+          Number(nowParts.day) + dayOffset,
+          hour,
+          minute,
+        ),
+      );
+      let candidate = targetDay;
+      const targetMs = targetDay.getTime();
+      for (let attempt = 0; attempt < 4; attempt += 1) {
+        const actual = wallClock(candidate);
+        const actualMs = Date.UTC(
+          Number(actual.year),
+          Number(actual.month) - 1,
+          Number(actual.day),
+          Number(actual.hour),
+          Number(actual.minute),
+        );
+        const adjustment = targetMs - actualMs;
+        if (adjustment === 0) break;
+        candidate = new Date(candidate.getTime() + adjustment);
+      }
+      return candidate;
+    };
+    const sameDay = buildRetryAt(0);
+    return sameDay.getTime() > now.getTime() ? sameDay : buildRetryAt(1);
+  } catch {
+    return null;
+  }
+}
@@ -116,6 +116,7 @@ import { budgetService } from "../budgets.js";
 import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js";
 import { issueRecoveryActionService } from "../issue-recovery-actions.js";
 import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
+import { parseProviderQuotaClockReset } from "../provider-quota-reset.js";
 import { hasUnverifiedNativeWorkspaceFinalizationOwner } from "../native-runtime/native-workspace-finalization-recovery-state.js";
 import {
  legacyExecutionNeedsReconciliationWithEvidence,
@@ -588,83 +589,6 @@ export type AdapterFailureRecoveryClassification =
  | { kind: "configuration_incomplete" }
  | null;

-function parseProviderQuotaClockReset(error: string, now: Date) {
-  const match = error.match(
-    /(?:try again at|resets?(?:\s+at)?)\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i,
-  );
-  if (!match) return null;
-
-  const hourValue = Number.parseInt(match[1] ?? "", 10);
-  const minute = Number.parseInt(match[2] ?? "0", 10);
-  const meridiem = (match[3] ?? "").toLowerCase();
-  if (!Number.isInteger(hourValue)) return null;
-  if (
-    meridiem ? hourValue < 1 || hourValue > 12 : hourValue < 0 || hourValue > 23
-  )
-    return null;
-  if (!Number.isInteger(minute) || minute < 0 || minute > 59) return null;
-
-  let hour = meridiem ? hourValue % 12 : hourValue;
-  if (meridiem === "p") hour += 12;
-  const timeZone = (match[4] ?? match[5])?.trim();
-  if (!timeZone) {
-    const retryAt = new Date(now);
-    retryAt.setUTCHours(hour, minute, 0, 0);
-    if (retryAt.getTime() <= now.getTime())
-      retryAt.setUTCDate(retryAt.getUTCDate() + 1);
-    return retryAt;
-  }
-
-  try {
-    const wallClock = (date: Date) =>
-      Object.fromEntries(
-        new Intl.DateTimeFormat("en-US", {
-          timeZone,
-          hourCycle: "h23",
-          year: "numeric",
-          month: "2-digit",
-          day: "2-digit",
-          hour: "2-digit",
-          minute: "2-digit",
-        })
-          .formatToParts(date)
-          .map((part) => [part.type, part.value]),
-      );
-    const nowParts = wallClock(now);
-    const buildRetryAt = (dayOffset: number) => {
-      const targetDay = new Date(
-        Date.UTC(
-          Number(nowParts.year),
-          Number(nowParts.month) - 1,
-          Number(nowParts.day) + dayOffset,
-          hour,
-          minute,
-        ),
-      );
-      let candidate = targetDay;
-      const targetMs = targetDay.getTime();
-      for (let attempt = 0; attempt < 4; attempt += 1) {
-        const actual = wallClock(candidate);
-        const actualMs = Date.UTC(
-          Number(actual.year),
-          Number(actual.month) - 1,
-          Number(actual.day),
-          Number(actual.hour),
-          Number(actual.minute),
-        );
-        const adjustment = targetMs - actualMs;
-        if (adjustment === 0) break;
-        candidate = new Date(candidate.getTime() + adjustment);
-      }
-      return candidate;
-    };
-    const sameDay = buildRetryAt(0);
-    return sameDay.getTime() > now.getTime() ? sameDay : buildRetryAt(1);
-  } catch {
-    return null;
-  }
-}
-
 export function classifyAdapterFailureForRecovery(
  latestRun: Pick<
    NonNullable<LatestIssueRun>,