mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts each agent run in a provider session on a remote sandbox > - A provider can refuse `session.open` because the account reached its rate limit or quota > - After one same-run retry, a repeat of that refusal ended the run as `native_provider_startup_rejected` > - That code blocks the task for a board Retry and sends the failure to Sentry, but the limit passes by itself > - This pull request records that refusal as a provider usage-limit hold, which already exists for limits on a turn > - The benefit is that Paperclip retries the task at the reset by itself, and Sentry gets no error for an expected wait ## Linked Issues or Issue Description Refs #20808, #19246, #18854. No public issue exists for this case. **What happened?** On staging, an account-wide Anthropic 429 refused `session.open` for all runs of one agent. Each run waited the one-hour default cooldown and retried on the same run. The retry met the same 429. The repeat check then ended each run as `native_provider_startup_rejected`. Sentry recorded 25 events in two bursts. Each task became blocked and waited for a person to click Retry. **What did you expect to happen?** A rate limit or quota at session start must wait for the reset, the same as a limit that the provider reports on a turn. The usage-limit hold retries the task at the reset by itself and does not report to Sentry. **Steps to reproduce** 1. Use a remote native runner with an ACPX provider whose account is rate limited. 2. Start a task run. The provider refuses `session.open` with `classification=provider_rate_limit` and a typed `providerFailure` (`phase: "startup"`, `category: "rate_limit"`). 3. Let the same-run retry run after the cooldown while the limit continues. 4. See that the run fails as `native_provider_startup_rejected`, the task becomes blocked for a board Retry, and Sentry gets a `run_failure` event. ## What Changed - `nativeRemoteStartupRejectionCode` returns `native_provider_usage_limit` instead of `native_provider_startup_rejected` when the typed startup failure is `rate_limit` or `quota_exhausted`. The first refusal with a proven runner exit still retries on the same run after the cooldown, as before. Other startup failures do not change. - `readNativeUsageLimitReset` also accepts a startup refusal when the run has no turn event. It reads the current failure (from the executor) or the run's saved `providerFailure`, and it requires `phase: "startup"`. A run with a turn event keeps the old rules. - `planProviderLimitResume` accepts the current startup failure, so the hold gets its reset before the run row records the failure. - The hold saves the startup refusal that created it (`providerLimitStartup`, with its observation time). When the hold resumes, it reads that refusal first. A same-run retry keeps the run row's first saved failure, so the run row can name an older reset. - Tests: a classifier unit test, an integration test for the hold and its automatic Retry, an executor-level test for two startup attempts with different resets, and a Sentry-report test for the startup shape. ## Verification - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "session.open the provider itself rejected"`: 9 tests pass. The new test fails on `master` with `Received: "native_provider_startup_rejected"`. - `npx vitest run src/services/native-runtime/native-provider-capacity.integration.test.ts src/services/__tests__/run-failure-report-native-usage-limit.test.ts`: 26 tests pass. The new integration test fails on `master` (the plan is not automatic without a turn event). - `npx vitest run` on `native-startup-rejected-retry`, `native-provider-capacity`, `native-startup-retry` and `native-provider-limit-cooldown` (integration), and `run-failure-report-native-usage-limit`: 76 tests pass. The new two-attempt test fails on the first head (`c3ee40f166`) with `reason: "reset_unknown"`. - The full `native-session-executor.test.ts`, `native-provider-limit-cooldown.integration.test.ts` and `run-failure-report*.test.ts` files: 1865 tests pass and 1 fails. The failure (`compares export process compatibility without changing conversation identity (changed: false)`) also fails on an unchanged `master` in the same sandbox. - I could not run the full server typecheck locally, because the sandbox has an 8 GiB memory limit and `tsc` stopped with an out-of-memory error. CI runs the typecheck. ## Risks - Low risk. The change applies only to a remote startup refusal with a typed `rate_limit` or `quota_exhausted` category that would otherwise end the run. - The task still shows as blocked during the hold, as for a limit on a turn. The hold now retries by itself at the reset, at most three times in a row without a successful run. - A quota without a reported reset gives the same "restore usage capacity" action as before, so a person still decides. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
667 lines
33 KiB
TypeScript
667 lines
33 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents,
|
|
agentWakeupRequests,
|
|
chatConversations,
|
|
chatEndpoints,
|
|
companies,
|
|
completionContracts,
|
|
createDb,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
issueRecoveryActions,
|
|
issues,
|
|
nativeRunFinalizations,
|
|
toolApplications,
|
|
toolConnections,
|
|
} from "@paperclipai/db";
|
|
import type {
|
|
ControlPlanePort,
|
|
NativeExecutionInputV1,
|
|
PrpEvent,
|
|
} from "@paperclipai/paperclip-runner";
|
|
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
|
|
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
|
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
|
...(await importOriginal<
|
|
typeof import("../../vendor/paperclip-runner/index.js")
|
|
>()),
|
|
executeNativeSession: provider.execute,
|
|
}));
|
|
|
|
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
|
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
|
import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js";
|
|
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
|
import {
|
|
planProviderLimitResume,
|
|
PROVIDER_LIMIT_RESUME_KEY_PREFIX,
|
|
readNativeUsageLimitReset,
|
|
registerProviderLimitResumeWork,
|
|
requestProviderLimitResumeRetry,
|
|
} from "./provider-limit-resume.js";
|
|
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
|
import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
|
|
|
|
describe("native provider capacity failure persistence", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase(
|
|
"native-provider-capacity-",
|
|
);
|
|
db = createDb(temporary.connectionString);
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Native capacity regression",
|
|
issuePrefix: "NPC",
|
|
});
|
|
await db.insert(agents).values({
|
|
id: agentId,
|
|
companyId,
|
|
name: "Native capacity agent",
|
|
status: "active",
|
|
adapterType: "paperclip_runner",
|
|
adapterConfig: { provider: "codex", model: "gpt-5.6-luna" },
|
|
});
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await temporary?.cleanup();
|
|
});
|
|
|
|
/** Runs one native session whose provider ends its turn with `payload`. */
|
|
async function failTurn(options: {
|
|
payload: Record<string, unknown>;
|
|
replayed?: boolean;
|
|
responsibleUserId?: string | null;
|
|
}) {
|
|
const issueId = randomUUID();
|
|
const runId = randomUUID();
|
|
const [issue] = await db
|
|
.insert(issues)
|
|
.values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Reply after a capacity failure",
|
|
status: "in_progress",
|
|
workMode: "standard",
|
|
assigneeAgentId: agentId,
|
|
})
|
|
.returning();
|
|
const [run] = await db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: runId,
|
|
companyId,
|
|
agentId,
|
|
status: "running",
|
|
runtimeMode: "native",
|
|
nativeIssueId: issueId,
|
|
invocationSource: "assignment",
|
|
triggerDetail: "system",
|
|
contextSnapshot: { issueId },
|
|
responsibleUserId: options.responsibleUserId ?? null,
|
|
})
|
|
.returning();
|
|
await db
|
|
.update(issues)
|
|
.set({ executionRunId: runId })
|
|
.where(eq(issues.id, issueId));
|
|
const native = await prepareNativeHeartbeatRun({
|
|
db,
|
|
run,
|
|
issue,
|
|
environmentLeaseId: randomUUID(),
|
|
});
|
|
const [contract] = await db
|
|
.select()
|
|
.from(completionContracts)
|
|
.where(eq(completionContracts.issueId, issueId));
|
|
const completionInput = buildNativeCompletionContract(issue, {
|
|
revision: contract.revision,
|
|
});
|
|
expect(completionInput).toEqual(contract.contractJson);
|
|
await db.insert(nativeRunFinalizations).values({
|
|
runId,
|
|
companyId,
|
|
issueId,
|
|
phase: "observed",
|
|
});
|
|
// A real terminal follows provider session establishment. Keep the
|
|
// recovery evidence explicit; this must not be a bootstrap retry.
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
sessionCheckpoint: {
|
|
providerSessionId: "capacity-test-provider-session",
|
|
providerIdentity: { threadId: "capacity-test-thread" },
|
|
},
|
|
},
|
|
})
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
|
|
const execution: NativeExecutionInputV1 = {
|
|
schema: "paperclip.native-execution-input.v1",
|
|
binding: {
|
|
companyId,
|
|
runId,
|
|
issueId,
|
|
agentId,
|
|
executionWorkspaceId: runId,
|
|
},
|
|
provider: { kind: "codex", model: "gpt-5.6-luna" },
|
|
task: {
|
|
identifier: issueId,
|
|
title: issue.title,
|
|
description: null,
|
|
prompt: "Reply to the current message.",
|
|
workMode: "standard",
|
|
},
|
|
workspace: {
|
|
cwd: "/tmp",
|
|
repoUrl: null,
|
|
repoRef: null,
|
|
branchName: null,
|
|
},
|
|
session: {
|
|
normalizedSessionId: native.normalizedSessionId,
|
|
driverKind: "codex_app_server",
|
|
protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
|
},
|
|
completionContract: {
|
|
id: contract.id,
|
|
sha256: contract.canonicalSha256,
|
|
schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: completionInput,
|
|
},
|
|
interactionResponses: [],
|
|
credentialBindings: [],
|
|
};
|
|
const event: PrpEvent = {
|
|
schema: "paperclip.prp.event.v1",
|
|
runId,
|
|
normalizedSessionId: native.normalizedSessionId,
|
|
turnId: native.turnId,
|
|
sourceInstanceId: native.runnerInstanceId,
|
|
sourceEventId: "capacity-terminal",
|
|
sourceSeq: 1,
|
|
sourceKind: "runner",
|
|
eventType: "turn.failed",
|
|
schemaVersion: 1,
|
|
priority: 0,
|
|
emittedAt: new Date().toISOString(),
|
|
payload: options.payload,
|
|
};
|
|
const openInput = {
|
|
identity: {
|
|
companyId,
|
|
issueId,
|
|
runId,
|
|
agentId,
|
|
sessionId: native.normalizedSessionId,
|
|
},
|
|
backendKind: "mock" as const,
|
|
sourceInstanceId: native.runnerInstanceId,
|
|
};
|
|
if (options.replayed) {
|
|
// Model a controller crash after the event commit but before its
|
|
// observational callback or failure coordinator could run.
|
|
const priorPort = new PaperclipControlPlanePort(db, {
|
|
companyId,
|
|
issueId,
|
|
runId,
|
|
agentId,
|
|
sessionId: native.normalizedSessionId,
|
|
sourceInstanceId: native.runnerInstanceId,
|
|
controlPlaneSourceInstanceId: "prior-controller",
|
|
completionContractId: contract.id,
|
|
completionContractSha256: contract.canonicalSha256,
|
|
});
|
|
await priorPort.openRun(openInput);
|
|
expect((await priorPort.appendEvent(event)).disposition).toBe(
|
|
"committed",
|
|
);
|
|
}
|
|
provider.execute
|
|
.mockReset()
|
|
.mockImplementation(
|
|
async (executeOptions: { controlPlane: ControlPlanePort }) => {
|
|
await executeOptions.controlPlane.openRun(openInput);
|
|
const appended = await executeOptions.controlPlane.appendEvent(event);
|
|
expect(appended.disposition).toBe(
|
|
options.replayed ? "duplicate" : "committed",
|
|
);
|
|
// Production reports the provider terminal through the event, not by
|
|
// pattern-matching this exception's arbitrary message.
|
|
throw new Error("provider turn stopped");
|
|
},
|
|
);
|
|
await expect(
|
|
executePaperclipNativeSession({
|
|
db,
|
|
execution,
|
|
runnerInstanceId: native.runnerInstanceId,
|
|
backend: {
|
|
descriptor: async () => {
|
|
throw new Error(
|
|
"Unexpected provider descriptor call in capacity test",
|
|
);
|
|
},
|
|
openSession: async () => {
|
|
throw new Error(
|
|
"Unexpected provider session call in capacity test",
|
|
);
|
|
},
|
|
},
|
|
}),
|
|
).rejects.toThrow("provider turn stopped");
|
|
expect(provider.execute).toHaveBeenCalledTimes(1);
|
|
return { runId, issueId, sourceEventId: event.sourceEventId };
|
|
}
|
|
|
|
async function holdFor(issueId: string) {
|
|
const [recovery] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
|
return recovery;
|
|
}
|
|
|
|
it.each([false, true])(
|
|
"stops automatic retries for a durable usage terminal (replayed=%s)",
|
|
async (replayed) => {
|
|
const { runId, issueId, sourceEventId } = await failTurn({
|
|
replayed,
|
|
payload: {
|
|
status: "failed",
|
|
error: {
|
|
codexErrorInfo: "usageLimitExceeded",
|
|
message: "Private account metadata",
|
|
},
|
|
},
|
|
});
|
|
const [coordinator] = await db
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(eq(nativeRunFinalizations.runId, runId));
|
|
expect(coordinator).toMatchObject({
|
|
phase: "terminal_failure",
|
|
failureCode: "native_provider_usage_limit",
|
|
recoveryState: "blocked",
|
|
attempt: 1,
|
|
nextAttemptAt: null,
|
|
leaseOwner: null,
|
|
});
|
|
const [failedRun] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
expect(failedRun.errorCode).toBe("native_provider_usage_limit");
|
|
const recovery = await holdFor(issueId);
|
|
expect(recovery).toMatchObject({
|
|
ownerType: "board",
|
|
cause: "native_provider_usage_limit",
|
|
wakePolicy: null,
|
|
});
|
|
expect(recovery.nextAction).toContain(
|
|
"Restore model provider usage capacity",
|
|
);
|
|
expect(recovery.nextAction).not.toContain("Private account metadata");
|
|
const [waitingIssue] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId));
|
|
expect(waitingIssue.status).toBe("blocked");
|
|
const events = await db
|
|
.select()
|
|
.from(heartbeatRunEvents)
|
|
.where(eq(heartbeatRunEvents.runId, runId));
|
|
expect(
|
|
events.filter((row) => row.sourceEventId === sourceEventId),
|
|
).toHaveLength(1);
|
|
// No reset was reported, so nothing retries the task by itself.
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: recovery.id }))
|
|
.resolves.toEqual({ requested: false, reason: "reset_unknown" });
|
|
},
|
|
);
|
|
|
|
/** The ACP harness's weekly-limit terminal, as staging recorded it (PAP-1057). */
|
|
function weeklyLimit(resetAt: Date) {
|
|
const month = resetAt.toLocaleString("en-US", { month: "short", timeZone: "UTC" });
|
|
const hour = resetAt.getUTCHours();
|
|
const clock = `${hour % 12 || 12}${hour < 12 ? "am" : "pm"}`;
|
|
return {
|
|
status: "failed",
|
|
error: {
|
|
code: "RUNTIME",
|
|
message: "ACP agent reported a terminal limit failure.",
|
|
providerFailure: {
|
|
schema: "paperclip.provider-failure.v1",
|
|
provider: "anthropic",
|
|
phase: "turn",
|
|
category: "quota_exhausted",
|
|
retryability: "requires_action",
|
|
message: `ACP agent reported a terminal limit failure.\nYou've hit your weekly limit · resets ${month} ${resetAt.getUTCDate()}, ${clock} (UTC)`,
|
|
},
|
|
},
|
|
};
|
|
}
|
|
|
|
it("waits for a reported usage-limit reset and then requests the task's Retry once", async () => {
|
|
const resetAt = new Date(Date.now() + 2 * 24 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const userId = `user-${randomUUID()}`;
|
|
const { runId, issueId } = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: userId });
|
|
|
|
const hold = await holdFor(issueId);
|
|
expect(hold).toMatchObject({ cause: "native_provider_usage_limit",
|
|
evidence: expect.objectContaining({ providerLimitResetAt: resetAt.toISOString() }) });
|
|
// The hold says when work resumes, not that its session state is ambiguous.
|
|
expect(hold.nextAction).toBe(`The provider usage limit resets at ${resetAt.toISOString()}. `
|
|
+ "Paperclip will retry this task then; Retry starts it sooner if usage capacity is already restored.");
|
|
|
|
// A hold saved by the previous release names no reset and calls the state ambiguous.
|
|
const { providerLimitResetAt: _reset, ...oldEvidence } = hold.evidence;
|
|
await db.update(issueRecoveryActions).set({ evidence: oldEvidence, nextAction: "Inspect the original provider failure "
|
|
+ "and explicitly resolve the ambiguous session state; do not open a replacement provider session." })
|
|
.where(eq(issueRecoveryActions.id, hold.id));
|
|
|
|
// Before the reset nothing starts; the owner keeps the reset as its deadline
|
|
// and tells the reader when work resumes.
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }))
|
|
.resolves.toEqual({ requested: false, waitUntil: resetAt });
|
|
expect(await holdFor(issueId)).toMatchObject({ nextAction: hold.nextAction,
|
|
evidence: expect.objectContaining({ providerLimitResetAt: resetAt.toISOString() }) });
|
|
|
|
// At the reset Paperclip asks for the same Retry the accountable user would.
|
|
const afterReset = new Date(resetAt.getTime() + 1000);
|
|
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, afterReset);
|
|
expect(outcome).toMatchObject({ requested: true });
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.idempotencyKey, `task-retry:provider-limit-reset:${runId}`)));
|
|
expect(receipt).toMatchObject({ reason: "task_retry", requestedByActorType: "user", requestedByActorId: userId,
|
|
status: "deferred_issue_execution", payload: expect.objectContaining({ issueId,
|
|
taskRetry: expect.objectContaining({ sourceRunId: runId, actorId: userId }) }) });
|
|
expect((await holdFor(issueId)).evidence.providerLimitResume).toMatchObject({ state: "requested",
|
|
requestId: receipt.id, resetAt: resetAt.toISOString() });
|
|
|
|
// A second pass never requests another Retry for the same hold.
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, afterReset))
|
|
.resolves.toEqual({ requested: false, reason: "not_waiting" });
|
|
const receipts = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry"),
|
|
eq(agentWakeupRequests.agentId, agentId)));
|
|
expect(receipts.filter(row => (row.payload as Record<string, unknown>).issueId === issueId)).toHaveLength(1);
|
|
});
|
|
|
|
it("backs off and retries a Claude account rate limit that reports no reset", async () => {
|
|
// Claude's account-level 429 says only "try again later": no window, no reset.
|
|
const payload = { status: "failed", error: { code: "RUNTIME", message: "ACP agent reported a terminal request failure.",
|
|
providerFailure: { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "turn", category: "rate_limit",
|
|
httpStatus: 429, retryability: "unknown",
|
|
message: "API error: 429 This request would exceed your account's rate limit. Please try again later" } } };
|
|
const userId = `user-${randomUUID()}`;
|
|
const failedAt = Date.now();
|
|
const { runId, issueId } = await failTurn({ payload, responsibleUserId: userId });
|
|
const hold = await holdFor(issueId);
|
|
// The hold waits the default cooldown that also holds the agent's new work.
|
|
const resetAt = new Date(String(hold.evidence.providerLimitResetAt));
|
|
expect(resetAt.getTime()).toBeGreaterThanOrEqual(failedAt + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS);
|
|
expect(resetAt.getTime()).toBeLessThanOrEqual(Date.now() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS);
|
|
expect(hold.nextAction).toBe(`The provider did not report when its rate limit resets. Paperclip will retry this task at ${
|
|
resetAt.toISOString()}; Retry starts it sooner if usage capacity is already restored.`);
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }))
|
|
.resolves.toEqual({ requested: false, waitUntil: resetAt });
|
|
// After the wait Paperclip asks for the Retry once; saved work resumes through Retry admission.
|
|
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, new Date(resetAt.getTime() + 1000));
|
|
expect(outcome).toMatchObject({ requested: true });
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.idempotencyKey, `task-retry:${PROVIDER_LIMIT_RESUME_KEY_PREFIX}${runId}`)));
|
|
expect(receipt).toMatchObject({ reason: "task_retry", requestedByActorId: userId });
|
|
});
|
|
|
|
it("retries a task whose session start the provider refused with a rate limit, before any turn", async () => {
|
|
// Staging PAPERCLIP-BACKEND-8Z: Claude's account 429 refused session.open, so no turn event exists.
|
|
const startupLimit = { schema: "paperclip.provider-failure.v1" as const, provider: "anthropic", phase: "startup" as const,
|
|
category: "rate_limit" as const, httpStatus: 429, retryability: "unknown" as const,
|
|
message: "API error: 429 This request would exceed your account's rate limit. Please try again later · model not changed" };
|
|
const userId = `user-${randomUUID()}`;
|
|
const issueId = randomUUID();
|
|
const runId = randomUUID();
|
|
const finishedAt = new Date();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Startup rate limit", status: "blocked",
|
|
workMode: "standard", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "failed", runtimeMode: "native",
|
|
nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId },
|
|
responsibleUserId: userId, errorCode: "native_provider_usage_limit", finishedAt,
|
|
resultJson: { providerFailure: startupLimit } });
|
|
const [hold] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId,
|
|
kind: "active_run_watchdog", ownerType: "board", cause: "native_provider_usage_limit",
|
|
fingerprint: `startup-limit:${runId}`, evidence: { runId }, nextAction: "Wait for the provider limit." }).returning();
|
|
|
|
const resetAt = new Date(finishedAt.getTime() + PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS);
|
|
// The executor plans from the failure it holds, before the run row records it.
|
|
await expect(planProviderLimitResume(db, runId, { failure: startupLimit, observedAt: finishedAt }))
|
|
.resolves.toEqual({ automatic: true, resetAt, actorId: userId, estimated: true });
|
|
// The hold owner later reads the same reset from the run's saved failure.
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }))
|
|
.resolves.toEqual({ requested: false, waitUntil: resetAt });
|
|
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, new Date(resetAt.getTime() + 1000));
|
|
expect(outcome).toMatchObject({ requested: true });
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.idempotencyKey, `task-retry:${PROVIDER_LIMIT_RESUME_KEY_PREFIX}${runId}`)));
|
|
expect(receipt).toMatchObject({ reason: "task_retry", requestedByActorId: userId });
|
|
|
|
// Only a refusal at session start counts without a turn: a saved turn-phase failure does not.
|
|
const turnRun = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: turnRun, companyId, agentId, status: "failed", runtimeMode: "native",
|
|
nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId },
|
|
errorCode: "native_provider_usage_limit", finishedAt, resultJson: { providerFailure: { ...startupLimit, phase: "turn" } } });
|
|
await expect(readNativeUsageLimitReset(db, { id: turnRun, companyId, resultJson: { providerFailure: { ...startupLimit, phase: "turn" } },
|
|
finishedAt })).resolves.toBeNull();
|
|
// A turn that started but never settled is still unknown, whatever the run's saved failure says.
|
|
await db.insert(heartbeatRunEvents).values({ companyId, runId: turnRun, agentId, seq: 1, eventType: "turn.started",
|
|
stream: "system", level: "info", message: "turn started" });
|
|
await expect(readNativeUsageLimitReset(db, { id: turnRun, companyId, resultJson: { providerFailure: startupLimit }, finishedAt },
|
|
{ failure: startupLimit, observedAt: finishedAt })).resolves.toBeNull();
|
|
});
|
|
|
|
it("starts the Retry from the hold's own deadline after a restart rebuilds it", async () => {
|
|
// A typed reset a few seconds out; the owner learns it only from durable rows.
|
|
const resetAt = new Date(Date.now() + 3000);
|
|
const payload = weeklyLimit(resetAt);
|
|
(payload.error.providerFailure as Record<string, unknown>).retryAt = resetAt.toISOString();
|
|
const userId = `user-${randomUUID()}`;
|
|
const { issueId } = await failTurn({ payload, responsibleUserId: userId });
|
|
const hold = await holdFor(issueId);
|
|
const errors: unknown[] = [];
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
|
onError: error => { errors.push(error); } });
|
|
const resume = vi.fn(async () => undefined);
|
|
try {
|
|
const worker = registerProviderLimitResumeWork(db, coordinator,
|
|
{ resume, schedulingSuppressed: async () => false }, () => true);
|
|
await worker.ready;
|
|
// Nothing starts before the reset.
|
|
expect(resume).not.toHaveBeenCalled();
|
|
await vi.waitFor(() => expect(resume).toHaveBeenCalledTimes(1), { timeout: 20_000, interval: 200 });
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry"),
|
|
eq(agentWakeupRequests.requestedByActorId, userId)));
|
|
expect(resume).toHaveBeenCalledWith(companyId, receipt.id);
|
|
expect(Date.now()).toBeGreaterThanOrEqual(resetAt.getTime());
|
|
expect((await holdFor(issueId)).evidence.providerLimitResume).toMatchObject({ state: "requested", requestId: receipt.id });
|
|
expect(hold.id).toBe((await holdFor(issueId)).id);
|
|
} finally {
|
|
await coordinator.stop();
|
|
}
|
|
expect(errors).toEqual([]);
|
|
}, 30_000);
|
|
|
|
it("leaves a visible Retry when no user is accountable for the run", async () => {
|
|
const resetAt = new Date(Date.now() + 3 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const { issueId } = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: null });
|
|
const hold = await holdFor(issueId);
|
|
expect(hold.nextAction).toBe(`The provider usage limit resets at ${resetAt.toISOString()}. `
|
|
+ "Retry the task after that time, or sooner if usage capacity is restored.");
|
|
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id },
|
|
new Date(resetAt.getTime() + 1000));
|
|
expect(outcome).toEqual({ requested: false, reason: "no_accountable_user" });
|
|
expect((await holdFor(issueId)).nextAction).toBe("Paperclip did not retry this task after the provider usage limit "
|
|
+ "reset because no user is accountable for the run. Retry the task when usage capacity is available.");
|
|
});
|
|
|
|
it("prepares a connected chat conversation's Retry through the chat service", async () => {
|
|
const resetAt = new Date(Date.now() + 2 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const userId = `user-${randomUUID()}`;
|
|
const { runId, issueId } = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: userId });
|
|
// The task belongs to a restricted external chat conversation.
|
|
const [application] = await db.insert(toolApplications).values({ companyId, name: "Slack", type: "chat" }).returning();
|
|
const connectionId = randomUUID();
|
|
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId: application!.id, uid: connectionId,
|
|
name: "Slack", transport: "chat_sdk", connectionPurpose: "channel" });
|
|
const endpointId = randomUUID();
|
|
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, provider: "slack", publicId: endpointId,
|
|
assignedAgentId: agentId, status: "active", externalExecutionPolicy: "restricted" });
|
|
await db.insert(chatConversations).values({ companyId, endpointId, issueId, externalConversationId: "channel",
|
|
externalThreadId: issueId, externalLabel: "#limits", state: "waiting" });
|
|
const hold = await holdFor(issueId);
|
|
const chatActionId = randomUUID();
|
|
const prepareChatRetry = vi.fn(async () => ({ actionId: chatActionId }));
|
|
const outcome = await requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id, prepareChatRetry },
|
|
new Date(resetAt.getTime() + 1000));
|
|
expect(outcome).toMatchObject({ requested: true });
|
|
expect(prepareChatRetry).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
|
companyId, issueId, agentId, failedRunId: runId, initiatedByUserId: userId }));
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.idempotencyKey, `task-retry:${PROVIDER_LIMIT_RESUME_KEY_PREFIX}${runId}`)));
|
|
expect(receipt.payload).toMatchObject({ taskRetry: expect.objectContaining({ chatActionId }) });
|
|
});
|
|
|
|
it("replaces the retry promise on every give-up path", async () => {
|
|
const resetAt = new Date(Date.now() + 2 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const afterReset = new Date(resetAt.getTime() + 1000);
|
|
// The failed run's terminal can no longer be read as a provider-observed reset.
|
|
const unreadable = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: `user-${randomUUID()}` });
|
|
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, unreadable.runId));
|
|
const unreadableHold = await holdFor(unreadable.issueId);
|
|
expect(unreadableHold.nextAction).toContain("Paperclip will retry this task then");
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: unreadableHold.id }, afterReset))
|
|
.resolves.toEqual({ requested: false, reason: "reset_unknown" });
|
|
expect((await holdFor(unreadable.issueId)).nextAction).toBe("Paperclip did not retry this task after the provider "
|
|
+ "usage limit reset because the provider's reset could no longer be read. Retry the task when usage capacity is available.");
|
|
// The hold's run no longer failed for the usage limit.
|
|
const changed = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: `user-${randomUUID()}` });
|
|
await db.update(heartbeatRuns).set({ errorCode: "adapter_failed" }).where(eq(heartbeatRuns.id, changed.runId));
|
|
const changedHold = await holdFor(changed.issueId);
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: changedHold.id }, afterReset))
|
|
.resolves.toEqual({ requested: false, reason: "source_run_mismatch" });
|
|
expect((await holdFor(changed.issueId)).nextAction).toBe("Paperclip did not retry this task after the provider "
|
|
+ "usage limit reset because the failed run no longer matches this hold. Retry the task when usage capacity is available.");
|
|
});
|
|
|
|
it("stops after three automatic Retries in a row and counts again after a successful run", async () => {
|
|
const resetAt = new Date(Date.now() + 2 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const afterReset = new Date(resetAt.getTime() + 1000);
|
|
const userId = `user-${randomUUID()}`;
|
|
const { issueId } = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: userId });
|
|
const hold = await holdFor(issueId);
|
|
// Earlier usage-limit Retries on this task, none followed by a success.
|
|
const earlierRetry = (task: string) => ({ companyId, agentId, source: "on_demand", triggerDetail: "manual",
|
|
reason: "task_retry", status: "completed", idempotencyKey: `task-retry:${PROVIDER_LIMIT_RESUME_KEY_PREFIX}${randomUUID()}`,
|
|
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: task },
|
|
requestedAt: new Date(Date.now() - 60_000) });
|
|
// Two in a row still leave this one.
|
|
await db.insert(agentWakeupRequests).values([earlierRetry(issueId), earlierRetry(issueId)]);
|
|
const [third] = await db.insert(agentWakeupRequests).values(earlierRetry(issueId)).returning();
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id }, afterReset))
|
|
.resolves.toEqual({ requested: false, reason: "automatic_retry_budget_used" });
|
|
expect((await holdFor(issueId)).nextAction).toBe("Paperclip did not retry this task after the provider usage limit "
|
|
+ "reset because three automatic Retries in a row did not restore usage. Retry the task when usage capacity is available.");
|
|
expect(third.id).toBeTruthy();
|
|
|
|
// A successful run on the task after earlier Retries restores the budget.
|
|
const second = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: userId });
|
|
await db.insert(agentWakeupRequests).values([earlierRetry(second.issueId), earlierRetry(second.issueId),
|
|
earlierRetry(second.issueId)]);
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", runtimeMode: "native",
|
|
nativeIssueId: second.issueId, invocationSource: "assignment", triggerDetail: "system",
|
|
contextSnapshot: { issueId: second.issueId }, finishedAt: new Date(Date.now() - 1000),
|
|
createdAt: new Date(Date.now() - 10 * 60_000) });
|
|
const secondHold = await holdFor(second.issueId);
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: secondHold.id }, afterReset))
|
|
.resolves.toMatchObject({ requested: true });
|
|
});
|
|
|
|
it("rebuilds every waiting hold after a restart, past one page", async () => {
|
|
const resetAt = new Date(Date.now() + 2 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const { issueId } = await failTurn({ payload: weeklyLimit(resetAt), responsibleUserId: `user-${randomUUID()}` });
|
|
const hold = await holdFor(issueId);
|
|
// A hold saved before holds named their reset; it sorts after 600 others.
|
|
const { providerLimitResetAt: _reset, ...oldEvidence } = hold.evidence;
|
|
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, hold.id));
|
|
const otherCompanyId = randomUUID();
|
|
await db.insert(companies).values({ id: otherCompanyId, name: "Other holds", issuePrefix: "OTH" });
|
|
const otherIssues = Array.from({ length: 600 }, () => randomUUID());
|
|
await db.insert(issues).values(otherIssues.map(id => ({ id, companyId: otherCompanyId, title: "Other", status: "in_progress" })));
|
|
const { id: _id, companyId: _company, sourceIssueId: _issue, createdAt: _c, updatedAt: _u, ...template } = hold;
|
|
for (let page = 0; page < 6; page++) {
|
|
await db.insert(issueRecoveryActions).values(Array.from({ length: 100 }, (_, index) => ({ ...template,
|
|
companyId: otherCompanyId, sourceIssueId: otherIssues[page * 100 + index]!,
|
|
id: `00000000-0000-4000-8000-${String(page * 100 + index).padStart(12, "0")}`,
|
|
evidence: { ...oldEvidence, runId: randomUUID() } })));
|
|
}
|
|
await db.insert(issueRecoveryActions).values({ ...hold, id: `ffffffff-ffff-4fff-bfff-${randomUUID().slice(-12)}`,
|
|
evidence: oldEvidence });
|
|
const errors: unknown[] = [];
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
|
onError: error => { errors.push(error); } });
|
|
try {
|
|
const worker = registerProviderLimitResumeWork(db, coordinator,
|
|
{ resume: vi.fn(async () => undefined), schedulingSuppressed: async () => false }, () => true);
|
|
await worker.ready;
|
|
// The hold past the first page reads its reset and records it.
|
|
await vi.waitFor(async () => expect((await holdFor(issueId)).evidence.providerLimitResetAt).toBe(resetAt.toISOString()),
|
|
{ timeout: 25_000, interval: 250 });
|
|
} finally {
|
|
await coordinator.stop();
|
|
}
|
|
expect(errors).toEqual([]);
|
|
}, 40_000);
|
|
|
|
it("reports a reset beyond the automatic wait without retrying by itself", async () => {
|
|
// A monthly limit, typed by the provider, three weeks out.
|
|
const resetAt = new Date(Date.now() + 21 * 24 * 60 * 60_000);
|
|
resetAt.setUTCMinutes(0, 0, 0);
|
|
const payload = weeklyLimit(resetAt);
|
|
(payload.error.providerFailure as Record<string, unknown>).retryAt = resetAt.toISOString();
|
|
const userId = `user-${randomUUID()}`;
|
|
const { issueId } = await failTurn({ payload, responsibleUserId: userId });
|
|
const hold = await holdFor(issueId);
|
|
expect(hold.nextAction).toBe(`The provider usage limit resets at ${resetAt.toISOString()}. `
|
|
+ "Retry the task after that time, or sooner if usage capacity is restored.");
|
|
expect(hold.evidence).toMatchObject({ providerLimitResetAt: resetAt.toISOString() });
|
|
await expect(requestProviderLimitResumeRetry(db, { companyId, actionId: hold.id },
|
|
new Date(resetAt.getTime() + 1000))).resolves.toEqual({ requested: false, reason: "reset_beyond_wait" });
|
|
expect((await holdFor(issueId)).nextAction).toBe(hold.nextAction);
|
|
const receipts = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.requestedByActorId, userId)));
|
|
expect(receipts).toEqual([]);
|
|
});
|
|
});
|