fix: hold an agent's new work while its provider usage limit is active (#19246)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:37:55 +01:00 · 2026-11-05 08:57:55 -06:00
1 parent d0afeacdd5
commit e37ab23dbc
@@ -2,7 +2,7 @@
 import { resolveGrokDisableWebSearch } from "../drivers/acpx/grok-web-search.js";
 import { redactObservabilityDiagnostic } from "../drivers/agent-observability.js";
 import { resolveClaudeEffort } from "../drivers/acpx/claude-effort.js";
-import { acpxProviderFailure, acpxTerminalResult } from "../drivers/acpx/provider-failure.js";
+import { acpxProviderFailure, acpxTerminalResult, type AcpxTerminalSessionFailure } from "../drivers/acpx/provider-failure.js";
 import { providerFailureFromError } from "../drivers/provider-failure.js";
 import { resolveGrokEffort } from "../drivers/acpx/grok-effort.js";
 import { parseProviderMode } from "../contracts/provider-mode.js";
@@ -426,6 +426,8 @@ async function dispatch(
    });
    activeCopilotEvidence = toolEvidence && "captureSemanticReceipt" in toolEvidence ? toolEvidence as CopilotToolEvidence : undefined;
    let usageBefore: unknown;
+    // ACPX reports a generic terminal error; the provider's typed reason arrives here first.
+    const sessionFailure: { value?: AcpxTerminalSessionFailure } = {};
    const timed = request.params.performance === true;
    const timingStart = receivedAt;
    let statusMs = 0;
@@ -442,12 +444,13 @@ async function dispatch(
          waitForInput(currentTurnId, providerRequest, context),
        onPermissionRequest: (providerRequest, context) =>
          waitForPermission(currentTurnId, activeAgent, providerRequest, context, toolEvidence),
+        onTerminalSessionFailure: (failure) => { sessionFailure.value = failure; },
      });
    } catch (error) {
      turnId = null;
      throw error;
    }
-    void pumpTurn(currentTurnId, runtimeTurn, activeHost, usageBefore, extensions.drain, activity, toolEvidence);
+    void pumpTurn(currentTurnId, runtimeTurn, activeHost, usageBefore, extensions.drain, activity, toolEvidence, sessionFailure);
    // Warm sessions may defer initialize until their first prompt. Publish only
    // the capabilities of that live initialized connection, never old disk state.
    const promptStart = timed ? performance.now() : 0;
@@ -664,6 +667,7 @@ async function pumpTurn(
  drainExtensions: () => Promise<void>,
  activity: AcpxActivityAdapter,
  toolEvidence?: AcpxToolEvidence,
+  sessionFailure: { value?: AcpxTerminalSessionFailure } = {},
 ): Promise<void> {
  let terminal: Record<string, unknown>;
  try {
@@ -716,7 +720,7 @@ async function pumpTurn(
      // not replace the provider's authoritative completed/cancelled result.
      diagnostic("acpx_terminal_usage_unavailable", safeMessage(error));
    }
-    terminal = openParams ? acpxTerminalResult(result, openParams.agent, process.env) : boundedSidecarValue(result);
+    terminal = openParams ? acpxTerminalResult(result, openParams.agent, process.env, sessionFailure.value) : boundedSidecarValue(result);
  } catch (error) {
    terminal = {
      status: "failed",
@@ -724,7 +728,7 @@ async function pumpTurn(
        ...(error instanceof AcpxApprovalRequiredError || error instanceof AcpxProviderProcessLostError ? { code: error.code } : {}),
        message: safeMessage(error),
        retryable: false,
-        ...(openParams ? { providerFailure: acpxProviderFailure(error, openParams.agent, "turn", process.env) ?? undefined } : {}),
+        ...(openParams ? { providerFailure: acpxProviderFailure(error, openParams.agent, "turn", process.env, false, sessionFailure.value) ?? undefined } : {}),
      },
    };
  } finally {
@@ -567,7 +567,10 @@ describe("Codex ACPX runtime adapter", () => {
    expect(JSON.stringify(vi.mocked(runtime.ensureSession).mock.calls)).not.toContain("clientCapabilities");
    const notification = vi.fn();
    const request = vi.fn(async (_method, _params, context) => ({ requestId: context.requestId }));
-    const admittedFirst = port.startTurn({ text: "First", requestId: "turn-1", onExtensionRequest: request, onExtensionNotification: notification });
+    const onTerminalSessionFailure = vi.fn();
+    const admittedFirst = port.startTurn({ text: "First", requestId: "turn-1", onExtensionRequest: request, onExtensionNotification: notification, onTerminalSessionFailure });
+    // ACPX reports only a generic terminal error; the typed reason must reach the host.
+    expect(vi.mocked(runtime.startTurn).mock.calls.at(-1)?.[0]).toMatchObject({ onTerminalSessionFailure });
    const signal = new AbortController().signal;
    await expect(created.onExtensionRequest!("cursor/ask_question", { sessionId: "backend-1" }, { requestId: 0, signal })).resolves.toEqual({ requestId: 0 });
    const sessionless = { toolCallId: "native-cursor-plan", plan: "Review the complete plan" };
@@ -1538,6 +1538,9 @@ function runtimePort(
          ...(input.onElicitation
            ? { onElicitation: input.onElicitation }
            : {}),
+          ...(input.onTerminalSessionFailure
+            ? { onTerminalSessionFailure: input.onTerminalSessionFailure }
+            : {}),
        });
      } catch (error) {
        releaseExtensionTurn();
@@ -30,33 +30,58 @@ const CATEGORIES = new Map<string, ProviderFailure["category"]>([
 // ACP reserves this JSON-RPC code for "Authentication required".
 const ACP_AUTH_REQUIRED = -32000;

+/** The typed AIR session failure ACPX reports beside its generic terminal error. */
+export type AcpxTerminalSessionFailure = { category: string; title?: string; details?: string };
+
+/**
+ * AIR's `limit` category also covers context, turn and configured budget
+ * limits. Only an exhausted account allowance is a provider limit; the
+ * provider's own title distinguishes it.
+ */
+function sessionLimitCategory(failure: AcpxTerminalSessionFailure | undefined): ProviderFailure["category"] | undefined {
+  if (failure?.category !== "limit") return undefined;
+  const text = [failure.title, failure.details].filter(value => typeof value === "string").join("\n");
+  if (/\brate[- ]limit/i.test(text)) return "rate_limit";
+  if (/no available quota|usage limit|you(?:'|’)ve hit your (?:\w+ )?limit|out of (?:extra )?usage|quota (?:limit )?exceeded/i.test(text)) {
+    return "quota_exhausted";
+  }
+  return undefined;
+}
+
 /**
 * `ownMessage` keeps an error's own message when it carries no ACP error. Use
 * it only where every rejection belongs to the provider: a failed terminal
- * result or a provider session ensure.
+ * result or a provider session ensure. `sessionFailure` adds the provider's
+ * title from a typed terminal session failure; its details stay private.
 */
 export function acpxProviderFailure(error: unknown, agent: string, phase: ProviderFailure["phase"],
-  environment: NodeJS.ProcessEnv = {}, ownMessage = false): ProviderFailure | null {
+  environment: NodeJS.ProcessEnv = {}, ownMessage = false, sessionFailure?: AcpxTerminalSessionFailure): ProviderFailure | null {
  const acp = acpError(error);
  const data = record(acp?.data);
  const detail = record(data.error);
-  const fallback = ownMessage ? record(error) : {};
-  const message = typeof detail.message === "string" ? detail.message
+  const fallback = ownMessage || sessionFailure ? record(error) : {};
+  const ownText = typeof detail.message === "string" ? detail.message
    : typeof data.details === "string" ? data.details
      : typeof acp?.message === "string" ? acp.message : fallback.message;
-  if (typeof message !== "string") return null;
+  const title = typeof sessionFailure?.title === "string" ? sessionFailure.title.trim() : "";
+  const message = typeof ownText === "string" && title && !ownText.includes(title) ? `${ownText}\n${title}`
+    : typeof ownText === "string" ? ownText : title || undefined;
+  if (typeof message !== "string" || !message) return null;
  const status = data.status ?? data.statusCode ?? detail.status
    ?? (agent === "claude" ? Number(/^API error: (\d{3})\b/.exec(message)?.[1]) : undefined);
  const code = detail.type ?? detail.code ?? data.errorType ?? data.errorKind;
  const category = (typeof code === "string" ? CATEGORIES.get(code) : undefined)
-    ?? (acp?.code === ACP_AUTH_REQUIRED ? "authentication" : undefined);
+    ?? (acp?.code === ACP_AUTH_REQUIRED ? "authentication" : undefined)
+    ?? sessionLimitCategory(sessionFailure);
  return createProviderFailure({ provider: agent === "claude" ? "anthropic" : agent === "codex" ? "openai" : agent,
    phase, message, httpStatus: status, providerCode: code, requestId: data.request_id, category, environment });
 }

-export function acpxTerminalResult(result: unknown, agent: string, environment: NodeJS.ProcessEnv = {}): Record<string, unknown> {
+export function acpxTerminalResult(result: unknown, agent: string, environment: NodeJS.ProcessEnv = {},
+  sessionFailure?: AcpxTerminalSessionFailure): Record<string, unknown> {
  const raw = record(result);
-  const providerFailure = raw.status === "failed" ? acpxProviderFailure(raw.error, agent, "turn", environment, true) : null;
+  const providerFailure = raw.status === "failed"
+    ? acpxProviderFailure(raw.error, agent, "turn", environment, true, sessionFailure) : null;
  const terminal = boundedSidecarValue(result);
  if (providerFailure) {
    terminal.status = "failed";
@@ -1778,6 +1778,7 @@ describe("ACPX runtime host", () => {
    const onElicitation = vi.fn();
    const onExtensionRequest = vi.fn();
    const onExtensionNotification = vi.fn();
+    const onTerminalSessionFailure = vi.fn();
    const runtime = runtimePort({ startTurn });
    const host = await AcpxRuntimeHost.open(
      {
@@ -1797,6 +1798,7 @@ describe("ACPX runtime host", () => {
        onElicitation,
        onExtensionRequest,
        onExtensionNotification,
+        onTerminalSessionFailure,
      }),
    ).toMatchObject({ requestId: turn.requestId, result: turn.result });
    expect(startTurn).toHaveBeenCalledWith({
@@ -1805,6 +1807,7 @@ describe("ACPX runtime host", () => {
      onElicitation,
      onExtensionRequest,
      onExtensionNotification,
+      onTerminalSessionFailure,
    });
    expect(() =>
      host.startTurn({ text: "Concurrent", requestId: "turn-2" }),
@@ -6,6 +6,7 @@ import { resolveClaudeEffort } from "./claude-effort.js";
 import { resolveGrokEffort } from "./grok-effort.js";
 import { piProviderConfiguration } from "./pi-provider-config.js";
 import { withAcpxTurnCancellation } from "./turn-cancellation.js";
+import type { AcpxTerminalSessionFailure } from "./provider-failure.js";
 import { resolveAcpxProviderMode } from "./provider-mode.js";
 import { resolvePiThinkingLevel, type PiThinkingLevel } from "./pi-thinking.js";
 import { preparedHarnessHome, preconfiguredHomeEnvironment, usesPreconfiguredHome } from "../preconfigured-home.js";
@@ -131,6 +132,8 @@ export interface AcpxRuntimeTurnInput {
  onPermissionRequest?: AcpRuntimeOptions["onPermissionRequest"];
  onExtensionRequest?: AcpRuntimeOptions["onExtensionRequest"];
  onExtensionNotification?: AcpRuntimeOptions["onExtensionNotification"];
+  /** Receives the typed AIR failure behind ACPX's generic terminal error. */
+  onTerminalSessionFailure?: (failure: AcpxTerminalSessionFailure) => void;
 }

 export interface AcpxRuntimeTurn {
@@ -914,6 +917,7 @@ export class AcpxRuntimeHost {
      ...(input.onPermissionRequest ? { onPermissionRequest: input.onPermissionRequest } : {}),
      ...(input.onExtensionRequest ? { onExtensionRequest: input.onExtensionRequest } : {}),
      ...(input.onExtensionNotification ? { onExtensionNotification: input.onExtensionNotification } : {}),
+      ...(input.onTerminalSessionFailure ? { onTerminalSessionFailure: input.onTerminalSessionFailure } : {}),
    });
    this.#activeTurn = turn;
    const managed = withAcpxTurnCancellation(turn, reason => this.close({ reason }), ACPX_TURN_CANCELLATION_SHUTDOWN_BOUND_MS);
@@ -56,6 +56,29 @@ describe("provider failure boundary", () => {
    expect(acpxProviderFailure(new Error(""), "claude", "startup", {}, true)).toBeNull();
  });

+  it("keeps a typed ACP session limit's provider title and classifies an exhausted allowance", () => {
+    const generic = "ACP agent reported a terminal limit failure.";
+    const failed = { status: "failed", error: { code: "RUNTIME", message: generic } };
+    const terminal = (title: string, category = "limit") =>
+      acpxTerminalResult(failed, "claude", { ANTHROPIC_API_KEY: "fixture-secret" }, { category, title, details: "private stack fixture-secret" });
+    expect(terminal("You've hit your session limit · resets 2am (UTC)")).toMatchObject({ status: "failed", error: { providerFailure: {
+      provider: "anthropic", phase: "turn", category: "quota_exhausted", retryability: "requires_action",
+      message: `${generic}\nYou've hit your session limit · resets 2am (UTC)` } } });
+    expect(terminal("The Claude account has no available quota.")).toMatchObject({ error: { providerFailure: { category: "quota_exhausted" } } });
+    expect(terminal("Claude is temporarily rate limited.")).toMatchObject({ error: { providerFailure: { category: "rate_limit" } } });
+    // Context, turn and configured budget limits are not an account allowance.
+    for (const title of ["This Claude turn reached its configured limit.", "This Claude session reached its configured budget.", "Context window limit exceeded"]) {
+      expect(terminal(title)).toMatchObject({ error: { providerFailure: { category: "unknown", message: `${generic}\n${title}` } } });
+    }
+    expect(terminal("You've hit your session limit", "request")).toMatchObject({ error: { providerFailure: { category: "unknown" } } });
+    expect(JSON.stringify(terminal("You've hit your session limit"))).not.toMatch(/fixture-secret|private stack/);
+    // Without the typed callback the generic terminal stays unclassified.
+    expect(acpxTerminalResult(failed, "claude")).toMatchObject({ error: { providerFailure: { category: "unknown", message: generic } } });
+    // The thrown form of the same terminal is classified the same way.
+    expect(acpxProviderFailure(new Error(generic), "claude", "turn", {}, false, { category: "limit", title: "Claude is temporarily rate limited." }))
+      .toMatchObject({ category: "rate_limit", message: `${generic}\nClaude is temporarily rate limited.` });
+  });
+
  it("keeps the ACPX terminal cause when the raw result exceeds its payload limit", () => {
    const terminal = acpxTerminalResult({ status: "failed", error: { ...acp,
      data: { ...acp.data, responseBody: "private-body".repeat(7000) } } }, "claude");