fix(native): back off and retry a provider rate limit that reports no reset (#22216)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - Native runs drive ACP agents, such as Claude, through the runner. The runner types each provider failure for the server.
> - When a provider limit stops a turn, the server holds the task. It retries the task automatically at the reset that the provider reports (#20692).
> - Claude can reject a request at the account rate limit with "This request would exceed your account's rate limit. Please try again later." This text has no reset time and no window.
> - The server then plans no automatic Retry. The task stays blocked until a person retries it, and the case work stops before a final reply.
> - This pull request gives a `rate_limit` failure without a reported reset the default provider-limit cooldown, and then requests the task's Retry.
> - The benefit is that a short account rate limit no longer strands a task. The existing Retry admission keeps the saved work and all gates.

## Linked Issues or Issue Description

Refs #20692, #19246, #21100

**What happened?**

A native task ran on a Claude agent. In the middle of the turn, Claude returned HTTP 429 with "This request would exceed your account's rate limit. Please try again later." The runner typed the failure as `rate_limit`, with no `retryAt`. The server put the task on hold with "Restore model provider usage capacity, then retry the task. The provider did not report when its usage limit resets." It did not plan an automatic Retry. The task stayed blocked until a person retried it.

**Expected behavior**

A rate limit passes by itself. Paperclip should wait a bounded time, and then retry the task. It should keep the saved work and should not start another request before the wait ends.

**Steps to reproduce**

1. Run a native task on a Claude agent whose account is at its rate limit.
2. Let Claude reject a request in the turn with the 429 account rate-limit text above.
3. See that the task hold has no reset time and that no automatic Retry is planned.

**Paperclip version or commit**

`master` at 8282d386da.

## What Changed

- `readNativeUsageLimitReset` (server, native runtime) now gives a `rate_limit` turn failure without a reported reset an estimated reset. The estimated reset is the failure time plus `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour). This is the same wait that already holds the agent's new work after the same failure, so both holds end at about the same time.
- The reset carries an `estimated` flag. The hold text says that the provider did not report the reset, and it gives the time when Paperclip will retry.
- A `quota_exhausted` failure without a reported reset does not change. It still waits for a person, because an exhausted allowance does not pass by itself.
- The existing bounds still apply: a maximum of three automatic Retries in a row without a successful run, Retry admission gates, and the accountable-user check.
- This change is a fallback. When the runner sets `retryAt` (for example, #21100 for a named rate-limit window) or the text contains a reset, Paperclip uses that time.

## Verification

- New test in `server/src/services/native-runtime/native-provider-capacity.integration.test.ts`: "backs off and retries a Claude account rate limit that reports no reset".
- Before the fix, the test failed: `AssertionError: expected NaN to be greater than or equal to 1795028502736`. The hold had no `providerLimitResetAt`, because no reset was planned.
- After the fix: `cd server && pnpm vitest run src/services/native-runtime/native-provider-capacity.integration.test.ts` passes (11 tests).
- Related files pass: `native-provider-limit-cooldown.integration.test.ts`, `src/__tests__/heartbeat-provider-limit-wait.test.ts`, `task-interruption-presentation.test.ts`, and the capacity file (4 files, 90 tests).
- `cd server && pnpm typecheck` passes.
- `pnpm verify` selected 2,925 of 3,792 Vitest files, because the changed module is deep in the server import graph. Its first selected group passed (2 files, 12 tests). I stopped the broad local run and rely on the required CI test checks for that coverage.

## Risks

- Low risk. The change applies only to a native turn failure that is typed `rate_limit` and has no reported reset.
- If the account stays at its limit for more than one hour, the automatic Retry can fail again. After three automatic Retries in a row, the task waits for a person, as before.
- The task stays in the blocked state during the wait, as it does for a reported reset. A person can select Retry to start it sooner.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: no subsystem doc describes this hold, and the hold text and tests describe the behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-18 15:24:52 -06:00
1 parent f472991647
commit c36e791bd9
2 files changed
+52 -7

No files matched your search

@@ -38,6 +38,7 @@ import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { PROVIDER_LIMIT_RESUME_KEY_PREFIX, registerProviderLimitResumeWork, requestProviderLimitResumeRetry } from "./provider-limit-resume.js";
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
describe("native provider capacity failure persistence", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -397,6 +398,32 @@ describe("native provider capacity failure persistence", () => {
expect(receipts.filter(row => (row.payload as Record<string, unknown>).issueId === issueId)).toHaveLength(1);
});
it("backs off and retries a Claude account rate limit that reports no reset", async () => {
// Claude's account-level 429 says only "try again later": no window, no reset.
const payload = { status: "failed", error: { code: "RUNTIME", message: "ACP agent reported a terminal request failure.",
providerFailure: { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "turn", category: "rate_limit",
httpStatus: 429, retryability: "unknown",
message: "API error: 429 This request would exceed your account's rate limit. Please try again later" } } };
const userId = `user-${randomUUID()}`;
const failedAt = Date.now();
const { runId, issueId } = await failTurn({ payload, responsibleUserId: userId });
const hold = await holdFor(issueId);
// The hold waits the default cooldown that also holds the agent's new work.
const resetAt = new Date(String(hold.evidence.providerLimitResetAt));
expect(resetAt.getTime()).toBeGreaterThanOrEqual(failedAt + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS);
expect(resetAt.getTime()).toBeLessThanOrEqual(Date.now() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS);
expect(hold.nextAction).toBe(`The provider did not report when its rate limit resets. Paperclip will retry this task at ${
resetAt.toISOString()}; Retry starts it sooner if usage capacity is already restored.`);
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }))
.resolves.toEqual({ requested: false, waitUntil: resetAt });
// After the wait Paperclip asks for the Retry once; saved work resumes through Retry admission.
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, new Date(resetAt.getTime() + 1000));
expect(outcome).toMatchObject({ requested: true });
const [receipt] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.idempotencyKey, `task-retry:${PROVIDER_LIMIT_RESUME_KEY_PREFIX}${runId}`)));
expect(receipt).toMatchObject({ reason: "task_retry", requestedByActorId: userId });
});
it("starts the Retry from the hold's own deadline after a restart rebuilds it", async () => {
// A typed reset a few seconds out; the owner learns it only from durable rows.
const resetAt = new Date(Date.now() + 3000);
@@ -15,6 +15,7 @@ import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import { isProvenTurnSettlement, TURN_SETTLED_EVENTS, TURN_STARTED_EVENTS } from "../heartbeat/restart-interrupted-turn.js";
import { parseProviderQuotaClockReset } from "../provider-quota-reset.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
import { requestTaskRetry } from "../task-retry.js";
/** The chat service's Retry callback, which a connected conversation's Retry needs. */
@@ -51,12 +52,16 @@ export type ProviderLimitReset = {
resetAt: Date; observedAt: Date;
/** False for a reset further out than Paperclip waits by itself (a monthly limit). */
withinAutomaticWait: boolean;
/** True when the provider reported no reset and Paperclip waits its default cooldown. */
estimated?: true;
};
/**
* When the provider said the usage limit resets, from the run's own durable
* events. Null unless the last turn event is a provider-observed settlement
* reporting an exhausted allowance with a reset after it.
* reporting an exhausted allowance with a reset after it. A rate limit that
* names no reset (Claude's "try again later") waits the default cooldown, the
* same wait that holds the agent's new work, so the task is not stranded.
*/
export async function readNativeUsageLimitReset(
db: Pick<Db, "select">,
@@ -74,10 +79,14 @@ export async function readNativeUsageLimitReset(
if (!failure || !LIMIT_CATEGORIES.includes(failure.category)) return null;
const observedAt = lastTurn.createdAt;
const typed = failure.retryAt ? new Date(failure.retryAt) : null;
const resetAt = typed && Number.isFinite(typed.getTime()) ? typed : parseProviderQuotaClockReset(failure.message, observedAt);
const reported = typed && Number.isFinite(typed.getTime()) ? typed : parseProviderQuotaClockReset(failure.message, observedAt);
// An exhausted allowance without a reset still needs a person; a rate limit passes by itself.
const estimated = !reported && failure.category === "rate_limit";
const resetAt = estimated ? new Date(observedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS) : reported;
if (!resetAt || resetAt.getTime() <= observedAt.getTime()) return null;
return { resetAt, observedAt,
withinAutomaticWait: resetAt.getTime() - observedAt.getTime() <= PROVIDER_LIMIT_RESUME_MAX_WAIT_MS };
withinAutomaticWait: resetAt.getTime() - observedAt.getTime() <= PROVIDER_LIMIT_RESUME_MAX_WAIT_MS,
...(estimated ? { estimated: true as const } : {}) };
}
/** The user whose authority Retry uses: the requester, else the run's accountable user. */
@@ -90,8 +99,8 @@ async function accountableUser(db: Pick<Db, "select">, run: typeof heartbeatRuns
}
export type ProviderLimitResumePlan =
| { automatic: true; resetAt: Date; actorId: string }
| { automatic: false; resetAt: Date | null };
| { automatic: true; resetAt: Date; actorId: string; estimated?: true }
| { automatic: false; resetAt: Date | null; estimated?: true };
/** Whether this failed run's task continues by itself once the limit resets. */
export async function planProviderLimitResume(db: Pick<Db, "select">, runId: string): Promise<ProviderLimitResumePlan> {
@@ -101,10 +110,18 @@ export async function planProviderLimitResume(db: Pick<Db, "select">, runId: str
if (!reset) return { automatic: false, resetAt: null };
// A reset beyond the bounded wait is still reported; only the automatic Retry is bounded.
const actorId = reset.withinAutomaticWait ? await accountableUser(db, run) : null;
return actorId ? { automatic: true, resetAt: reset.resetAt, actorId } : { automatic: false, resetAt: reset.resetAt };
const estimated = reset.estimated ? { estimated: true as const } : {};
return actorId ? { automatic: true, resetAt: reset.resetAt, actorId, ...estimated }
: { automatic: false, resetAt: reset.resetAt, ...estimated };
}
export function providerLimitNextAction(plan: ProviderLimitResumePlan) {
if (plan.estimated && plan.resetAt) {
const unreported = "The provider did not report when its rate limit resets.";
return plan.automatic
? `${unreported} Paperclip will retry this task at ${plan.resetAt.toISOString()}; Retry starts it sooner if usage capacity is already restored.`
: `${unreported} Retry the task after ${plan.resetAt.toISOString()}, or sooner if usage capacity is restored.`;
}
if (plan.automatic) {
return `The provider usage limit resets at ${plan.resetAt.toISOString()}. Paperclip will retry this task then; Retry starts it sooner if usage capacity is already restored.`;
}
@@ -187,7 +204,8 @@ export async function requestProviderLimitResumeRetry(db: Db, input: {
const actorId = await accountableUser(db, run);
await db.update(issueRecoveryActions).set({
nextAction: providerLimitNextAction(actorId
? { automatic: true, resetAt: reset.resetAt, actorId } : { automatic: false, resetAt: reset.resetAt }),
? { automatic: true, resetAt: reset.resetAt, actorId, estimated: reset.estimated }
: { automatic: false, resetAt: reset.resetAt, estimated: reset.estimated }),
evidence: sql`coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb) || jsonb_build_object('providerLimitResetAt', ${reset.resetAt.toISOString()}::text)`,
updatedAt: now,
}).where(and(eq(issueRecoveryActions.id, action.id), eq(issueRecoveryActions.status, "active")));