fix: hold an agent's new work while its provider usage limit is active (#19246)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Routine-driven agents run on the native Runner through ACP harnesses
such as Claude
> - When the agent's provider account runs out of allowance, every new
wake for that agent fails the same way
> - The ACP harness reported the exhausted allowance as a generic
"terminal limit failure" with an `unknown` category, so Paperclip could
not tell it from a context or turn limit
> - Paperclip recorded a reset time only for the failing run's own
startup retry; other wakes of the same agent kept dispatching, each
spent about two minutes on workspace setup, and then failed
> - This pull request classifies the limit, keeps its reset time, and
holds the agent's new work until that reset
> - The benefit is that a limited agent stops burning runs, shows a
clear "waiting for usage limit reset" state with a time, and resumes its
queued work after the reset

## Linked Issues or Issue Description

Refs #13148
Refs #13379

**What happened?**
A routine-driven Claude agent on the native Runner failed 29 times in
about 15 hours. 20 runs failed with `native_provider_terminal_failed`
and "ACP agent reported a terminal limit failure." The recorded provider
failure had category `unknown`. 6 runs failed with
`provider_session_open_rejected`. Their `session.open` result is
`command_execution_failed` with classification `provider_rate_limit`,
and the provider failure is HTTP 429 `rate_limit` ("would exceed your
account's rate limit"). The startup retry of one such run recorded a
cooldown until one hour later. A new wake for a different task of the
same agent dispatched 16 minutes later, spent two minutes on workspace
setup, and failed with the same 429. Separately, 4 runs failed with
"Project authentication conflicts with this agent's managed AI
connection". That message did not say which file or setting conflicted.
The same message was also shown when the check could not finish.

**What did you expect?**
A provider usage limit is classified distinctly. Paperclip records the
reset time when the provider reports one. New work for that agent waits
for the reset with a visible reason. Queued work resumes after the
reset, and no doomed run starts. A configuration conflict is reported
only when one is found, and the message names the file to fix.

**Steps to reproduce**
1. Use a native Runner agent with a Claude ACP harness whose account is
rate limited or out of usage.
2. Let a routine or assignment wake the agent on several tasks.
3. See each run fail with a generic terminal limit failure or a
`session.open` rejection, and see new runs keep starting.

## What Changed

- Runner: The ACPX turn now receives the typed AIR session failure
(`onTerminalSessionFailure`) through the runtime host and the ACPX port.
The sidecar adds the provider's title to the provider failure message.
Only a rate-limit or quota title becomes `rate_limit` or
`quota_exhausted`. Context, turn and configured budget limits stay
`unknown`. The failure details stay private.
- Server: A native `turn.failed` with a `rate_limit` or
`quota_exhausted` provider failure is now `native_provider_usage_limit`,
the same code as the Codex usage-limit terminal.
- Server: `agentProviderLimitCooldown` (extracted from the startup
retry) also reads a reset clock in the provider's wording, such as
"resets 2am (UTC)". The clock parser moved from the recovery service to
`provider-quota-reset.ts` without changes.
- Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace
or provider work, a queued run for an agent with an active provider
limit is cancelled as `provider_limit_wait`. The run records the reset
time, whether the provider reported it, the category and the run that
reported the limit. Paperclip schedules one retry of the same wake at
the reset through the existing AI connection wait lane. That lane does
not consume the failure retry budget. Manual runs, recovering runs,
conversation turns and pool-routed connections are not held.
- Server: The execution projection shows "Waiting for usage limit reset"
with the reset time and the next step.
- Server: Only a found project authentication conflict is a
configuration failure. Its message names the project file and the
settings to remove. A check that cannot finish (timeout or another exit
code) is now an ordinary failure, not a configuration conflict.
- Docs: `doc/execution-semantics.md` describes the agent-level limit
wait.

## Verification

Red before the fix, green after:
- `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An
automation wake after a 429 `session.open` rejection dispatched the
adapter before the fix. Now the run is cancelled as
`provider_limit_wait` and a scheduled retry exists at the reset. After
the reset passes, the retry is promoted and the work succeeds. A manual
run, a later success, expired evidence and an unclassified terminal
still dispatch.
- `native-startup-retry.integration.test.ts`: "reads the reset clock
from an ACP terminal limit failure's provider wording" (it fell back to
one hour before).
- `native-session-executor.test.ts`: An ACP `turn.failed` with a typed
`quota_exhausted` or `rate_limit` failure is a usage-limit terminal;
`unknown` is not.
- `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A
typed AIR limit title is kept and classified; context, turn and budget
limits are not; details do not leak.
- `execution-projection.test.ts`, `ai-connections.test.ts`: The wait
label, and conflict versus unfinished check.
- Safety cases: `runtime-host.test.ts` and
`codex-runtime-adapter.test.ts` forward the callback. The existing
startup retry, provider capacity, recovery classification, legacy
reconciliation, retry scheduling and execution configuration suites
pass.

Commands:
- `cd server && npx vitest run
src/__tests__/heartbeat-provider-limit-wait.test.ts
src/__tests__/ai-connections.test.ts src/services/native-runtime/
src/services/recovery/ src/services/execution-projection.test.ts
src/services/legacy-execution-recovery.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-execution-configuration.test.ts`
- `cd packages/paperclip-runner && npx vitest run
src/drivers/provider-failure.test.ts
src/drivers/acpx/runtime-host.test.ts
src/drivers/acpx/codex-runtime-adapter.test.ts
src/cli/acpx-runtime-sidecar.test.ts`
- `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and
provider-failure contract checks pass.

I did not run a live provider limit on staging. The live evidence is the
staging run records for the affected agent, which I read only.

## Risks

- The limit evidence is per agent, not per account. If the board changes
the agent's AI connection, automatic wakes can still wait until the old
reset (at most one hour when no reset was reported, and at most one
day). A manual run is not held, and one successful run clears the
evidence.
- A new wake that arrives during a limit is now cancelled and retried at
the reset, instead of failing. The task stays in its status, and the run
shows the reset time.
- The Runner change takes effect when the provider pack is rebuilt and
deployed. Until then, the server still holds new work using the existing
`rate_limit` evidence from `session.open` rejections.
- A turn terminal with a typed rate limit or quota is now
`native_provider_usage_limit` instead of
`native_provider_terminal_failed`. Both are terminal and board-owned.
The recovery text now asks the board to restore provider usage capacity.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging run
records) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/__tests__/ai-connections.test.ts

@@ -32,7 +32,7 @@ import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
 import * as runnerProbe from "../vendor/paperclip-runner/index.js";
 import * as nativeCodexRunner from "../services/native-runtime/native-codex-runner.js";
 import { testNativeRunnerAuthentication } from "../services/native-runtime/setup-readiness.js";
-import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy, managedAiSessionFingerprintConfig } from "../services/ai-connection-runtime.js";
+import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy, isAiConnectionIncompatible, managedAiSessionFingerprintConfig } from "../services/ai-connection-runtime.js";
 import { execute as executeGemini, testEnvironment as testGeminiEnvironment } from "@paperclipai/adapter-gemini-local/server";
 import { toolAccessService } from "../services/tool-access.js";
 import { secretService } from "../services/secrets.js";
@@ -1949,6 +1949,26 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello"
       await expect(assertManagedAiProjectAuth({ args: ["--api-key=override"] }, "xai", target)).rejects.toThrow("overrides");
     } finally { execute.mockRestore(); }
   });
+  it("reports only a found conflict as configuration, naming the files to fix", async () => {
+    const execute = vi.spyOn(executionTarget, "runAdapterExecutionTargetProcess");
+    const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace/project" } as Parameters<typeof assertManagedAiProjectAuth>[2];
+    const outcome = (exitCode: number | null, timedOut = false) =>
+      ({ exitCode, stdout: "", stderr: "", signal: null, timedOut }) as Awaited<ReturnType<typeof executionTarget.runAdapterExecutionTargetProcess>>;
+    try {
+      execute.mockResolvedValue(outcome(42));
+      const conflict = await assertManagedAiProjectAuth({}, "anthropic", target).catch((error: unknown) => error);
+      expect(isAiConnectionIncompatible(conflict)).toBe(true);
+      expect(String((conflict as Error).message)).toContain(".claude/settings.json");
+      // A check that could not complete is not proof of a configuration conflict.
+      for (const failed of [outcome(43), outcome(null, true), outcome(127)]) {
+        execute.mockResolvedValue(failed);
+        const unavailable = await assertManagedAiProjectAuth({}, "anthropic", target).catch((error: unknown) => error);
+        expect(unavailable).toBeInstanceOf(Error);
+        expect(isAiConnectionIncompatible(unavailable)).toBe(false);
+      }
+      expect(isAiConnectionIncompatible(await assertManagedAiProjectAuth({ args: ["--settings=x"] }, "anthropic", target).catch((error: unknown) => error))).toBe(true);
+    } finally { execute.mockRestore(); }
+  });
   it("requires a typed project authentication capability for task-only environments", async () => {
     const execute = vi.spyOn(executionTarget, "runAdapterExecutionTargetProcess");
     const description = { kind: "description", taskId: "task", projectAuthCheckVersion: 1,

server/src/__tests__/heartbeat-provider-limit-wait.test.ts

@@ -0,0 +1,197 @@
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db";
+import { registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
+import { heartbeatService } from "../services/heartbeat.js";
+import { createProviderLimitWait } from "../services/heartbeat/provider-limit-wait.js";
+import { executionProjectionsForRuns } from "../services/execution-projection.js";
+import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
+
+const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
+if (!recoveryPostgresSupport.supported) console.warn(`Skipping provider limit wait tests: ${recoveryPostgresSupport.reason}`);
+
+/**
+ * An agent whose provider account is out of allowance must not start new work
+ * that will fail the same way. The wake waits for the reported reset instead.
+ */
+describePostgres("heartbeat provider usage limit wait", () => {
+  let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
+  let db: ReturnType<typeof createDb>;
+  let heartbeat: ReturnType<typeof heartbeatService>;
+  const adapterType = "codex_local";
+  const dispatched = new Set<string>();
+  const oldHome = process.env.PAPERCLIP_HOME;
+  let home: string;
+
+  beforeAll(async () => {
+    temporary = await startRecoveryPostgres("heartbeat-provider-limit-");
+    db = createDb(temporary.connectionString);
+    home = await mkdtemp(path.join(tmpdir(), "paperclip-provider-limit-"));
+    process.env.PAPERCLIP_HOME = home;
+    heartbeat = heartbeatService(db);
+    registerServerAdapter({
+      type: adapterType, supportsLocalAgentJwt: false,
+      execute: async input => {
+        dispatched.add(input.runId);
+        await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId)));
+        return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };
+      },
+      testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }),
+    });
+  }, 60_000);
+
+  afterAll(async () => {
+    await heartbeat?.drainActiveRunExecutions();
+    unregisterServerAdapter(adapterType);
+    await db?.$client.end({ timeout: 0 });
+    await temporary?.cleanup();
+    if (oldHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = oldHome;
+    if (home) await rm(home, { recursive: true, force: true });
+  });
+
+  async function seed(prior: Array<Partial<typeof heartbeatRuns.$inferInsert>>) {
+    const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
+    const userId = `limit-user-${companyId}`;
+    await db.insert(authUsers).values({ id: userId, name: "Limit user", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
+    await db.insert(companies).values({ id: companyId, name: "Provider limit fixture", issuePrefix: `L${companyId.slice(0, 6)}` });
+    await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Limited agent", adapterType, status: "idle", adapterConfig: {} });
+    await db.insert(issues).values({ id: issueId, companyId, title: "Triage the next alert", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1 });
+    for (const values of prior) {
+      await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", triggerDetail: "system", ...values });
+    }
+    return { companyId, agentId, issueId };
+  }
+
+  async function wake(f: Awaited<ReturnType<typeof seed>>, manual = false) {
+    const run = await heartbeat.wakeup(f.agentId, manual
+      ? { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } }
+      : { source: "automation", triggerDetail: "system", reason: "issue_assigned", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
+    expect(run).not.toBeNull();
+    await heartbeat.drainActiveRunExecutions();
+    const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
+    return finished;
+  }
+
+  const rateLimited = (minutesAgo: number, overrides: Record<string, unknown> = {}) => ({
+    status: "failed",
+    startedAt: new Date(Date.now() - (minutesAgo + 2) * 60_000),
+    finishedAt: new Date(Date.now() - minutesAgo * 60_000),
+    errorCode: "provider_session_open_rejected",
+    resultJson: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
+      category: "rate_limit", httpStatus: 429, retryability: "unknown",
+      message: "API error: 429 This request would exceed your account's rate limit. Please try again later", ...overrides } },
+  });
+
+  it("defers new work until the reported reset instead of dispatching a doomed run", async () => {
+    const limited = rateLimited(16);
+    const f = await seed([limited]);
+    const resetAt = new Date(limited.finishedAt.getTime() + 60 * 60_000);
+    const finished = await wake(f);
+
+    expect(dispatched.has(finished.id)).toBe(false);
+    expect(finished).toMatchObject({
+      status: "cancelled",
+      errorCode: "provider_limit_wait",
+      resultJson: {
+        executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false },
+        providerLimitWait: { resetAt: resetAt.toISOString(), resetReported: false, category: "rate_limit" },
+      },
+    });
+    expect(finished.error).toContain(resetAt.toISOString());
+    const successors = await db.select().from(heartbeatRuns)
+      .where(eq(heartbeatRuns.agentId, f.agentId));
+    const retry = successors.find(run => run.status === "scheduled_retry");
+    expect(retry).toMatchObject({ scheduledRetryReason: "ai_connection_pool_wait" });
+    expect(retry!.scheduledRetryAt!.getTime()).toBeGreaterThanOrEqual(resetAt.getTime());
+    expect(retry!.scheduledRetryAt!.getTime()).toBeLessThan(resetAt.getTime() + 5 * 60_000);
+    const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id));
+    expect(events.some(event => event.message?.includes("provider usage limit"))).toBe(true);
+    const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    expect(issue.status).not.toBe("blocked");
+    // The persisted projection names the wait and its reset.
+    expect((await executionProjectionsForRuns(db, f.companyId, [retry!.id])).get(retry!.id)).toMatchObject({
+      label: "Waiting for usage limit reset", phase: "retry_scheduled",
+      nextAction: expect.stringContaining(retry!.scheduledRetryAt!.toISOString()),
+    });
+
+    // Nothing is dropped: the wait starts the work once the reset has passed.
+    expect((await heartbeat.promoteDueScheduledRetries(new Date())).runIds).not.toContain(retry!.id);
+    // Model the hour passing by moving the limit evidence back past its reset.
+    await db.update(heartbeatRuns).set({ startedAt: new Date(Date.now() - 3 * 60 * 60_000), finishedAt: new Date(Date.now() - 2 * 60 * 60_000) })
+      .where(eq(heartbeatRuns.errorCode, "provider_session_open_rejected"));
+    expect((await heartbeat.promoteDueScheduledRetries(new Date(retry!.scheduledRetryAt!.getTime() + 1))).runIds).toContain(retry!.id);
+    await heartbeat.resumeQueuedRuns();
+    await heartbeat.drainActiveRunExecutions();
+    expect(dispatched.has(retry!.id)).toBe(true);
+    const [resumed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retry!.id));
+    expect(resumed.status).toBe("succeeded");
+  }, 60_000);
+
+  it("releases the task when its retry cannot be scheduled, even if diagnostics fail", async () => {
+    const f = await seed([rateLimited(16)]);
+    const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
+    const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
+      invocationSource: "automation", triggerDetail: "system", contextSnapshot: { issueId: f.issueId } }).returning();
+    const deps = {
+      setRunStatusIfRunning: vi.fn(async () => ({ updated: true, run: { ...run, status: "cancelled" } })),
+      setWakeupStatus: vi.fn(async () => { throw new Error("wakeup write failed"); }),
+      appendRunEvent: vi.fn(async () => { throw new Error("event write failed"); }),
+      scheduleBoundedRetryForRun: vi.fn(async () => { throw new Error("retry write failed"); }),
+      releaseIssueExecutionAndPromote: vi.fn(async () => undefined),
+      finalizeAgentStatus: vi.fn(async () => undefined),
+    };
+    const { deferRunForProviderLimit } = createProviderLimitWait(db, deps as never);
+    await expect(deferRunForProviderLimit(run, agent, { resumingAdmittedRun: false, conversation: false })).resolves.toBe(true);
+    expect(deps.scheduleBoundedRetryForRun).toHaveBeenCalledTimes(2);
+    expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledOnce();
+    expect(deps.finalizeAgentStatus).toHaveBeenCalledOnce();
+
+    // A scheduled retry keeps the task lock for its successor.
+    deps.scheduleBoundedRetryForRun.mockReset().mockResolvedValue({ outcome: "scheduled" } as never);
+    deps.releaseIssueExecutionAndPromote.mockClear();
+    await expect(deferRunForProviderLimit(run, agent, { resumingAdmittedRun: false, conversation: false })).resolves.toBe(true);
+    expect(deps.releaseIssueExecutionAndPromote).not.toHaveBeenCalled();
+  });
+
+  it("recovers a wait whose retry was lost to a crash after its cancellation committed", async () => {
+    const f = await seed([]);
+    const [lost] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "cancelled",
+      invocationSource: "automation", triggerDetail: "system", issueId: f.issueId, scopeKind: "issue",
+      startedAt: new Date(Date.now() - 10 * 60_000), finishedAt: new Date(Date.now() - 9 * 60_000), errorCode: "provider_limit_wait",
+      resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } },
+      contextSnapshot: { issueId: f.issueId } }).returning();
+    await db.update(issues).set({ status: "in_progress", executionRunId: lost.id, executionLockedAt: new Date(Date.now() - 9 * 60_000) })
+      .where(eq(issues.id, f.issueId));
+    await heartbeat.sweepStaleIssueLocks();
+    await heartbeat.reconcileStrandedAssignedIssues();
+    await heartbeat.drainActiveRunExecutions();
+    const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agentId));
+    // Existing stranded-work recovery gives the wake a bounded successor; the gate re-checks the limit when it runs.
+    expect(runs.find(run => run.id !== lost.id)).toMatchObject({ status: "scheduled_retry", retryOfRunId: lost.id });
+    const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    expect(issue).toMatchObject({ status: "in_progress", executionRunId: null });
+  }, 60_000);
+
+  it("still dispatches a manual run, and work after a later success or an expired reset", async () => {
+    const manual = await seed([rateLimited(16)]);
+    expect(dispatched.has((await wake(manual, true)).id)).toBe(true);
+
+    const recovered = await seed([rateLimited(30), {
+      status: "succeeded", startedAt: new Date(Date.now() - 10 * 60_000), finishedAt: new Date(Date.now() - 9 * 60_000),
+    }]);
+    expect(dispatched.has((await wake(recovered)).id)).toBe(true);
+
+    const expired = await seed([rateLimited(90)]);
+    expect(dispatched.has((await wake(expired)).id)).toBe(true);
+
+    const unrelated = await seed([{ ...rateLimited(5), resultJson: { providerFailure: {
+      schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "turn", category: "unknown",
+      retryability: "unknown", message: "ACP agent reported a terminal limit failure." } } }]);
+    expect(dispatched.has((await wake(unrelated)).id)).toBe(true);
+  }, 60_000);
+});