From 7485e01161d3ee45dc823cd6c510f017a559cc32 Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 04:29:05 -0800 Subject: [PATCH] fix(heartbeat): keep a credential denial an owner action when the lease lapses during AI selection (#22379) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat prepares each run before the agent starts. One preparation step selects the AI account for the run. > - AI selection can reject an account for a reason that only its owner can fix. An example is a credential that is not shared with the responsible user. > - Run setup gives a lost controller lease priority over every other failure code. When the lease lapsed during the same setup, the run got `controller_lease_lost` instead of `configuration_incomplete`. > - `controller_lease_lost` is retryable. So the run retried and failed again with the same rejection. The code also bypassed the Sentry filter for owned AI selection blockers, so the server sent a spurious Sentry event. > - This pull request keeps `configuration_incomplete` when AI selection itself rejected the account, even when the lease is lost at the same time. > - The benefit is that the owner gets the correct action at once, with no wasted retry and no Sentry noise. ## Linked Issues or Issue Description Refs #15714 (this PR adds the Sentry filter for owned AI selection blockers, and this change reuses that check). **What happened?** A legacy run was in setup. AI selection correctly rejected the account (`ai_connection_credential_not_shared`). At the same time, a lease renewal stalled for about 29 seconds and the controller lease lapsed. Setup recorded the run as `controller_lease_lost` with the configuration message. The run was retried. The retry failed again as `configuration_incomplete`. The first failure also went to Sentry, because the filter only matches `configuration_incomplete`. **Expected behavior** When AI selection rejects the account for an owned reason, the run fails as `configuration_incomplete`. The run is not retried. The board gets the configuration action. No Sentry event is sent. The run result keeps the lease-loss evidence. **Steps to reproduce** 1. Start a legacy run whose AI account selection fails with an owned rejection (for example, a credential that is not shared with the responsible user). 2. Make the controller lease lapse while setup selects the AI account. 3. Look at the run `errorCode`, the retry schedule, and the Sentry report. **Paperclip version or commit** `master` at c41145e72a. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - Added `isOwnedAiConnectionSelectionRejection` in `server/src/lib/ai-connection-configuration-failure.ts`. It matches a `configurationIncomplete` result with reason `ai_connection_unavailable` and a known `selectionFailure`. - `run-completion.ts` setup failure: a lost lease no longer replaces the error code when AI selection itself rejected the account. Any other AI preparation error (the broad wrapper) still records a retryable `controller_lease_lost`. - `run-failure-report.ts`: the Sentry filter now uses the same shared check. Its behavior does not change. - Added tests: - A unit test in `run-completion.test.ts` for both cases (owned rejection keeps its code; unclassified error keeps `controller_lease_lost`). - A full heartbeat test in `heartbeat-process-recovery.test.ts`. It checks the error code, no retry, the board configuration action, and no Sentry report. - A test in `run-failure-report.test.ts` for the shared check. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/services/__tests__/run-failure-report.test.ts` - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "credential sharing denial"` - Without the `run-completion.ts` change, both new regression tests fail with `controller_lease_lost`. With the change, they pass. - In the author's sandbox, 4 sandbox-provider lease tests in `run-completion.test.ts` and 4 provider-outage tests in `heartbeat-process-recovery.test.ts` fail. They fail the same way on untouched `master` in that sandbox, so the cause is the environment. CI runs the full suite. ## Risks - Low risk. The change applies only when AI selection recorded an owned rejection. A lost lease with any other setup error keeps the same retryable code as before. - The run is no longer retried in this case. That is correct, because every host repeats the same rejection until the owner acts. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip Co-authored-by: Claude Opus 5.5 --- .../heartbeat-process-recovery.test.ts | 48 +++++++++++++++++++ .../ai-connection-configuration-failure.ts | 11 +++++ .../__tests__/run-failure-report.test.ts | 14 ++++++ .../ai-connection-configuration-failure.ts | 2 +- .../services/heartbeat/run-completion.test.ts | 17 +++++++ .../src/services/heartbeat/run-completion.ts | 7 ++- server/src/services/run-failure-report.ts | 6 +-- 7 files changed, 99 insertions(+), 6 deletions(-) diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 8422a92ce12..1cde5010c5c 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -1787,6 +1787,54 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { }, ); + it("routes a credential sharing denial to its owner when the controller lease lapses during AI selection", async () => { + const { companyId, agentId, issueId, runId } = await seedQueuedIssueRunFixture(); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "responsible-user", status: "active", membershipRole: "member" }).onConflictDoNothing(); + await db.update(agents).set({ runtimeConfig: { + heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 }, + aiConnection: { provider: "openai", method: "api_key", mode: "responsible_user" }, + } }).where(eq(agents.id, agentId)); + let runController: AbortController | undefined; + const originalWatch = controllerLeases.watchLegacyControllerLease; + const watcher = vi.spyOn(controllerLeases, "watchLegacyControllerLease").mockImplementation((...args) => { + if (args[1].id === runId) runController = args[2]; + return originalWatch(...args); + }); + // The renewal deadline passes while selection reads the grant, then the + // selection's own sharing denial reaches the setup failure path. + const prepare = vi.spyOn(aiConnectionRuntime, "prepareManagedAiRuntime").mockImplementationOnce(async () => { + runController!.abort(new controllerLeases.LegacyControllerLeaseLostError("renewal_deadline", + { sinceRenewalMs: 37_725, renewalPendingMs: 29_418, deadlineLateMs: 52, renewalCount: 2 })); + throw aiConnectionCredentialNotSharedFailure(); + }); + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRuns(); + await waitForRunToSettle(heartbeat, runId, 5_000); + await heartbeat.waitForRunExecutionDrain(runId); + await waitForPendingRunFailureReports(); + expect(prepare).toHaveBeenCalledOnce(); + expect(await heartbeat.getRun(runId)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete", + error: "This credential is not shared with the responsible user", resultJson: { + configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_credential_not_shared" }, + controllerLeaseLoss: { cause: "renewal_deadline", renewalPendingMs: 29_418 }, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + // No retry that would repeat the same denial before the owner acts. + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + expect(issue).toMatchObject({ status: "blocked", executionRunId: null }); + const [action] = await selectRecoveryActionsForIssue(issueId); + expect(action).toMatchObject({ status: "active", kind: "configuration_validation", cause: "configuration_incomplete", ownerType: "board" }); + } finally { + await heartbeat.drainActiveRunExecutions().catch(() => undefined); + prepare.mockRestore(); + watcher.mockRestore(); + await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId)); + } + }); + it.each(["timeout", "upstream", "cleanup_pending", "cleanup_pending_edited", "cleanup_pending_exhausted", "edited", "exhausted", "new_message", "no_claim", "reassigned", "superseded", "stopped"])( "settles explicit retry admission through real executor cleanup: %s", async scenario => { const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture(); diff --git a/server/src/lib/ai-connection-configuration-failure.ts b/server/src/lib/ai-connection-configuration-failure.ts index 5acb51c6d62..28bdc0da2d7 100644 --- a/server/src/lib/ai-connection-configuration-failure.ts +++ b/server/src/lib/ai-connection-configuration-failure.ts @@ -22,3 +22,14 @@ export function readAiConnectionConfigurationFailure(error: unknown): AiConnecti export function isAiConnectionConfigurationReason(value: unknown): value is AiConnectionConfigurationReason { return typeof value === "string" && reasons.some((reason) => reason === value); } + +/** + * A run's `configurationIncomplete` result records an AI selection that an owned + * producer rejected, not the preparation step's broad catch wrapper. The same + * selection fails again on any host until its owner acts. + */ +export function isOwnedAiConnectionSelectionRejection(configurationIncomplete: unknown): boolean { + if (configurationIncomplete === null || typeof configurationIncomplete !== "object" || Array.isArray(configurationIncomplete)) return false; + const configuration = configurationIncomplete as Record; + return configuration.reason === "ai_connection_unavailable" && isAiConnectionConfigurationReason(configuration.selectionFailure); +} diff --git a/server/src/services/__tests__/run-failure-report.test.ts b/server/src/services/__tests__/run-failure-report.test.ts index 9e288030a26..ce6a396bd58 100644 --- a/server/src/services/__tests__/run-failure-report.test.ts +++ b/server/src/services/__tests__/run-failure-report.test.ts @@ -724,6 +724,20 @@ describeEmbeddedPostgres("reportRunFailure", () => { expect(run).toEqual(saved); }); + it.each(["configuration_incomplete", "controller_lease_lost"] as const)( + "decides a credential denial that coincided with a lease loss by its recorded code: %s", async (errorCode) => { + await seedCompanyAndAgent(); + const run = aiSelectionRun({ errorCode, error: "This credential is not shared with the responsible user" }); + Object.assign(run.resultJson!, { + controllerLeaseLoss: { cause: "renewal_deadline", renewalCount: 2, renewalPendingMs: 29_418 }, + cancellation: { source: "controller_lease", expected: false, reason: "Legacy controller lease lost (renewal_deadline)" }, + }); + (run.resultJson!.configurationIncomplete as Record).selectionFailure = "ai_connection_credential_not_shared"; + await reportRunFailure(db, run, { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledTimes(errorCode === "controller_lease_lost" ? 1 : 0); + }, + ); + it.each([undefined, null, {}, [], "", "database_error", "ai_connection_busy"])( "reports AI connection wrappers without a known selection cause: %j", async (selectionFailure) => { await seedCompanyAndAgent(); diff --git a/server/src/services/ai-connection-configuration-failure.ts b/server/src/services/ai-connection-configuration-failure.ts index a31275b3185..b59214d2ed9 100644 --- a/server/src/services/ai-connection-configuration-failure.ts +++ b/server/src/services/ai-connection-configuration-failure.ts @@ -1,6 +1,6 @@ import { forbidden, unprocessable } from "../errors.js"; import { recordAiConnectionConfigurationFailure, type AiConnectionConfigurationReason } from "../lib/ai-connection-configuration-failure.js"; -export { readAiConnectionConfigurationFailure, isAiConnectionConfigurationReason } from "../lib/ai-connection-configuration-failure.js"; +export { readAiConnectionConfigurationFailure, isAiConnectionConfigurationReason, isOwnedAiConnectionSelectionRejection } from "../lib/ai-connection-configuration-failure.js"; type CredentialAccessFailure = { connectionName: string; grantId?: string }; const credentialAccessFailures = new WeakMap(); diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index 219b630feaa..a2f603c9ea4 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -1865,6 +1865,23 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)( resultJson: { cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_deadline)" } } }); }); + it.each([ + { name: "a rejected AI selection keeps its owner action", selectionFailure: "ai_connection_credential_not_shared", errorCode: "configuration_incomplete" }, + { name: "an unclassified AI preparation error stays a retryable lease loss", selectionFailure: undefined, errorCode: "controller_lease_lost" }, + ] as const)("records a lease lost during AI selection: $name", async ({ selectionFailure, errorCode }) => { + const f = await fixture(); + f.controller.abort(new LegacyControllerLeaseLostError("renewal_deadline", + { sinceRenewalMs: 37_725, renewalPendingMs: 29_418, deadlineLateMs: 52, renewalCount: 2 })); + await f.completion.failRunSetup(f.setup(new ConfigurationIncompleteFailure("This credential is not shared with the responsible user", { + configurationIncomplete: { reason: "ai_connection_unavailable", ...(selectionFailure ? { selectionFailure } : {}) } }))); + expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode, + error: "This credential is not shared with the responsible user", resultJson: { + configurationIncomplete: { reason: "ai_connection_unavailable" }, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, + controllerLeaseLoss: { cause: "renewal_deadline", renewalPendingMs: 29_418, renewalCount: 2 }, + cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_deadline)" } } }); + }); + it("does not make a lease loss retryable once provider work has started", async () => { const f = await fixture(); f.controller.abort(new LegacyControllerLeaseLostError("renewal_error")); diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index c76c9010cec..67b57d35159 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -34,6 +34,7 @@ import { isWorkspaceValidationFailedRun, isWorkspaceValidationFailure, attachPap import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../legacy-workspace-restore-recovery.js"; import { cancellationResultJson, controllerLeaseLossCancellation, readRunCancellation } from "../run-cancellation.js"; import { CONTROLLER_LEASE_LOST_FAILURE_CODE, LegacyControllerLeaseLostError } from "../legacy-controller-lease.js"; +import { isOwnedAiConnectionSelectionRejection } from "../ai-connection-configuration-failure.js"; import { REVIEWED_CHAT_BINDING_NOT_READY, ReviewedChatBindingNotReadyError, reviewedChatBindingDenialResultJson } from "../reviewed-chat-binding-diagnostics.js"; import { reportRunFailure } from "../run-failure-report.js"; import { accountRunCost } from "../run-cost-accounting.js"; @@ -1674,8 +1675,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC ? outerErr : null; const runnerRevisionMismatch = !providerDispatchStarted && run.runtimeMode === "legacy" && run.nativeSessionId === null && run.runnerInstanceId === null && isRunnerRevisionMismatch(outerErr); + // A lease lost while AI selection rejected the account does not make the + // run retryable: every host repeats that rejection until its owner acts. + const ownedAiSelectionRejection = isOwnedAiConnectionSelectionRejection( + configurationIncompleteSetupFailure?.resultJson.configurationIncomplete); const setupFailureErrorCode = - (controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ?? + (controllerLeaseLost && !ownedAiSelectionRejection ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ?? (storageUnavailable ? STORAGE_UNAVAILABLE_CODE : null) ?? remoteWorkspacePreparationTimeout?.code ?? reusableSandboxResumeFailure?.code ?? diff --git a/server/src/services/run-failure-report.ts b/server/src/services/run-failure-report.ts index e4592e70901..0e6314c7c58 100644 --- a/server/src/services/run-failure-report.ts +++ b/server/src/services/run-failure-report.ts @@ -14,7 +14,7 @@ import { import { logger } from "../middleware/logger.js"; import { isUnexpectedRunCancellation } from "./run-cancellation.js"; import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; -import { isAiConnectionConfigurationReason } from "./ai-connection-configuration-failure.js"; +import { isOwnedAiConnectionSelectionRejection } from "./ai-connection-configuration-failure.js"; import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js"; import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js"; import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js"; @@ -88,10 +88,8 @@ function isAiConnectionSelectionBlocker(run: HeartbeatRun, options: RunFailureRe run.executionStage !== "preparing" || options.phase !== "setup" || run.exitCode != null || run.signal != null ) return false; - const configuration = asRecord(run.resultJson?.configurationIncomplete); const recovery = asRecord(run.resultJson?.executionRecovery); - return configuration?.reason === "ai_connection_unavailable" && - isAiConnectionConfigurationReason(configuration.selectionFailure) && + return isOwnedAiConnectionSelectionRejection(run.resultJson?.configurationIncomplete) && recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false; }