fix(heartbeat): keep a credential denial an owner action when the lease lapses during AI selection (#22379)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat prepares each run before the agent starts. One
preparation step selects the AI account for the run.
> - AI selection can reject an account for a reason that only its owner
can fix. An example is a credential that is not shared with the
responsible user.
> - Run setup gives a lost controller lease priority over every other
failure code. When the lease lapsed during the same setup, the run got
`controller_lease_lost` instead of `configuration_incomplete`.
> - `controller_lease_lost` is retryable. So the run retried and failed
again with the same rejection. The code also bypassed the Sentry filter
for owned AI selection blockers, so the server sent a spurious Sentry
event.
> - This pull request keeps `configuration_incomplete` when AI selection
itself rejected the account, even when the lease is lost at the same
time.
> - The benefit is that the owner gets the correct action at once, with
no wasted retry and no Sentry noise.

## Linked Issues or Issue Description

Refs #15714 (this PR adds the Sentry filter for owned AI selection
blockers, and this change reuses that check).

**What happened?**
A legacy run was in setup. AI selection correctly rejected the account
(`ai_connection_credential_not_shared`). At the same time, a lease
renewal stalled for about 29 seconds and the controller lease lapsed.
Setup recorded the run as `controller_lease_lost` with the configuration
message. The run was retried. The retry failed again as
`configuration_incomplete`. The first failure also went to Sentry,
because the filter only matches `configuration_incomplete`.

**Expected behavior**
When AI selection rejects the account for an owned reason, the run fails
as `configuration_incomplete`. The run is not retried. The board gets
the configuration action. No Sentry event is sent. The run result keeps
the lease-loss evidence.

**Steps to reproduce**
1. Start a legacy run whose AI account selection fails with an owned
rejection (for example, a credential that is not shared with the
responsible user).
2. Make the controller lease lapse while setup selects the AI account.
3. Look at the run `errorCode`, the retry schedule, and the Sentry
report.

**Paperclip version or commit**
`master` at c41145e72a.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- Added `isOwnedAiConnectionSelectionRejection` in
`server/src/lib/ai-connection-configuration-failure.ts`. It matches a
`configurationIncomplete` result with reason `ai_connection_unavailable`
and a known `selectionFailure`.
- `run-completion.ts` setup failure: a lost lease no longer replaces the
error code when AI selection itself rejected the account. Any other AI
preparation error (the broad wrapper) still records a retryable
`controller_lease_lost`.
- `run-failure-report.ts`: the Sentry filter now uses the same shared
check. Its behavior does not change.
- Added tests:
- A unit test in `run-completion.test.ts` for both cases (owned
rejection keeps its code; unclassified error keeps
`controller_lease_lost`).
- A full heartbeat test in `heartbeat-process-recovery.test.ts`. It
checks the error code, no retry, the board configuration action, and no
Sentry report.
  - A test in `run-failure-report.test.ts` for the shared check.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/__tests__/run-failure-report.test.ts`
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "credential sharing
denial"`
- Without the `run-completion.ts` change, both new regression tests fail
with `controller_lease_lost`. With the change, they pass.
- In the author's sandbox, 4 sandbox-provider lease tests in
`run-completion.test.ts` and 4 provider-outage tests in
`heartbeat-process-recovery.test.ts` fail. They fail the same way on
untouched `master` in that sandbox, so the cause is the environment. CI
runs the full suite.

## Risks

- Low risk. The change applies only when AI selection recorded an owned
rejection. A lost lease with any other setup error keeps the same
retryable code as before.
- The run is no longer retried in this case. That is correct, because
every host repeats the same rejection until the owner acts.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 04:29:05 -08:00
1 parent 812e30d423
commit 7485e01161
7 files changed
+99 -6

No files matched your search

@@ -1787,6 +1787,54 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
},
);
it("routes a credential sharing denial to its owner when the controller lease lapses during AI selection", async () => {
const { companyId, agentId, issueId, runId } = await seedQueuedIssueRunFixture();
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "responsible-user", status: "active", membershipRole: "member" }).onConflictDoNothing();
await db.update(agents).set({ runtimeConfig: {
heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 },
aiConnection: { provider: "openai", method: "api_key", mode: "responsible_user" },
} }).where(eq(agents.id, agentId));
let runController: AbortController | undefined;
const originalWatch = controllerLeases.watchLegacyControllerLease;
const watcher = vi.spyOn(controllerLeases, "watchLegacyControllerLease").mockImplementation((...args) => {
if (args[1].id === runId) runController = args[2];
return originalWatch(...args);
});
// The renewal deadline passes while selection reads the grant, then the
// selection's own sharing denial reaches the setup failure path.
const prepare = vi.spyOn(aiConnectionRuntime, "prepareManagedAiRuntime").mockImplementationOnce(async () => {
runController!.abort(new controllerLeases.LegacyControllerLeaseLostError("renewal_deadline",
{ sinceRenewalMs: 37_725, renewalPendingMs: 29_418, deadlineLateMs: 52, renewalCount: 2 }));
throw aiConnectionCredentialNotSharedFailure();
});
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
await waitForRunToSettle(heartbeat, runId, 5_000);
await heartbeat.waitForRunExecutionDrain(runId);
await waitForPendingRunFailureReports();
expect(prepare).toHaveBeenCalledOnce();
expect(await heartbeat.getRun(runId)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete",
error: "This credential is not shared with the responsible user", resultJson: {
configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_credential_not_shared" },
controllerLeaseLoss: { cause: "renewal_deadline", renewalPendingMs: 29_418 },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
expect(mockAdapterExecute).not.toHaveBeenCalled();
// No retry that would repeat the same denial before the owner acts.
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(issue).toMatchObject({ status: "blocked", executionRunId: null });
const [action] = await selectRecoveryActionsForIssue(issueId);
expect(action).toMatchObject({ status: "active", kind: "configuration_validation", cause: "configuration_incomplete", ownerType: "board" });
} finally {
await heartbeat.drainActiveRunExecutions().catch(() => undefined);
prepare.mockRestore();
watcher.mockRestore();
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
}
});
it.each(["timeout", "upstream", "cleanup_pending", "cleanup_pending_edited", "cleanup_pending_exhausted", "edited", "exhausted", "new_message", "no_claim", "reassigned", "superseded", "stopped"])(
"settles explicit retry admission through real executor cleanup: %s", async scenario => {
const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture();
@@ -22,3 +22,14 @@ export function readAiConnectionConfigurationFailure(error: unknown): AiConnecti
export function isAiConnectionConfigurationReason(value: unknown): value is AiConnectionConfigurationReason {
return typeof value === "string" && reasons.some((reason) => reason === value);
}
/**
* A run's `configurationIncomplete` result records an AI selection that an owned
* producer rejected, not the preparation step's broad catch wrapper. The same
* selection fails again on any host until its owner acts.
*/
export function isOwnedAiConnectionSelectionRejection(configurationIncomplete: unknown): boolean {
if (configurationIncomplete === null || typeof configurationIncomplete !== "object" || Array.isArray(configurationIncomplete)) return false;
const configuration = configurationIncomplete as Record<string, unknown>;
return configuration.reason === "ai_connection_unavailable" && isAiConnectionConfigurationReason(configuration.selectionFailure);
}
@@ -724,6 +724,20 @@ describeEmbeddedPostgres("reportRunFailure", () => {
expect(run).toEqual(saved);
});
it.each(["configuration_incomplete", "controller_lease_lost"] as const)(
"decides a credential denial that coincided with a lease loss by its recorded code: %s", async (errorCode) => {
await seedCompanyAndAgent();
const run = aiSelectionRun({ errorCode, error: "This credential is not shared with the responsible user" });
Object.assign(run.resultJson!, {
controllerLeaseLoss: { cause: "renewal_deadline", renewalCount: 2, renewalPendingMs: 29_418 },
cancellation: { source: "controller_lease", expected: false, reason: "Legacy controller lease lost (renewal_deadline)" },
});
(run.resultJson!.configurationIncomplete as Record<string, unknown>).selectionFailure = "ai_connection_credential_not_shared";
await reportRunFailure(db, run, { phase: "setup" });
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(errorCode === "controller_lease_lost" ? 1 : 0);
},
);
it.each([undefined, null, {}, [], "", "database_error", "ai_connection_busy"])(
"reports AI connection wrappers without a known selection cause: %j", async (selectionFailure) => {
await seedCompanyAndAgent();
@@ -1,6 +1,6 @@
import { forbidden, unprocessable } from "../errors.js";
import { recordAiConnectionConfigurationFailure, type AiConnectionConfigurationReason } from "../lib/ai-connection-configuration-failure.js";
export { readAiConnectionConfigurationFailure, isAiConnectionConfigurationReason } from "../lib/ai-connection-configuration-failure.js";
export { readAiConnectionConfigurationFailure, isAiConnectionConfigurationReason, isOwnedAiConnectionSelectionRejection } from "../lib/ai-connection-configuration-failure.js";
type CredentialAccessFailure = { connectionName: string; grantId?: string };
const credentialAccessFailures = new WeakMap<Error, CredentialAccessFailure>();
@@ -1865,6 +1865,23 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
resultJson: { cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_deadline)" } } });
});
it.each([
{ name: "a rejected AI selection keeps its owner action", selectionFailure: "ai_connection_credential_not_shared", errorCode: "configuration_incomplete" },
{ name: "an unclassified AI preparation error stays a retryable lease loss", selectionFailure: undefined, errorCode: "controller_lease_lost" },
] as const)("records a lease lost during AI selection: $name", async ({ selectionFailure, errorCode }) => {
const f = await fixture();
f.controller.abort(new LegacyControllerLeaseLostError("renewal_deadline",
{ sinceRenewalMs: 37_725, renewalPendingMs: 29_418, deadlineLateMs: 52, renewalCount: 2 }));
await f.completion.failRunSetup(f.setup(new ConfigurationIncompleteFailure("This credential is not shared with the responsible user", {
configurationIncomplete: { reason: "ai_connection_unavailable", ...(selectionFailure ? { selectionFailure } : {}) } })));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode,
error: "This credential is not shared with the responsible user", resultJson: {
configurationIncomplete: { reason: "ai_connection_unavailable" },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
controllerLeaseLoss: { cause: "renewal_deadline", renewalPendingMs: 29_418, renewalCount: 2 },
cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_deadline)" } } });
});
it("does not make a lease loss retryable once provider work has started", async () => {
const f = await fixture();
f.controller.abort(new LegacyControllerLeaseLostError("renewal_error"));
@@ -34,6 +34,7 @@ import { isWorkspaceValidationFailedRun, isWorkspaceValidationFailure, attachPap
import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../legacy-workspace-restore-recovery.js";
import { cancellationResultJson, controllerLeaseLossCancellation, readRunCancellation } from "../run-cancellation.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE, LegacyControllerLeaseLostError } from "../legacy-controller-lease.js";
import { isOwnedAiConnectionSelectionRejection } from "../ai-connection-configuration-failure.js";
import { REVIEWED_CHAT_BINDING_NOT_READY, ReviewedChatBindingNotReadyError, reviewedChatBindingDenialResultJson } from "../reviewed-chat-binding-diagnostics.js";
import { reportRunFailure } from "../run-failure-report.js";
import { accountRunCost } from "../run-cost-accounting.js";
@@ -1674,8 +1675,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
? outerErr : null;
const runnerRevisionMismatch = !providerDispatchStarted && run.runtimeMode === "legacy"
&& run.nativeSessionId === null && run.runnerInstanceId === null && isRunnerRevisionMismatch(outerErr);
// A lease lost while AI selection rejected the account does not make the
// run retryable: every host repeats that rejection until its owner acts.
const ownedAiSelectionRejection = isOwnedAiConnectionSelectionRejection(
configurationIncompleteSetupFailure?.resultJson.configurationIncomplete);
const setupFailureErrorCode =
(controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
(controllerLeaseLost && !ownedAiSelectionRejection ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
(storageUnavailable ? STORAGE_UNAVAILABLE_CODE : null) ??
remoteWorkspacePreparationTimeout?.code ??
reusableSandboxResumeFailure?.code ??
+2 -4
View File
@@ -14,7 +14,7 @@ import {
import { logger } from "../middleware/logger.js";
import { isUnexpectedRunCancellation } from "./run-cancellation.js";
import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure";
import { isAiConnectionConfigurationReason } from "./ai-connection-configuration-failure.js";
import { isOwnedAiConnectionSelectionRejection } from "./ai-connection-configuration-failure.js";
import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js";
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js";
@@ -88,10 +88,8 @@ function isAiConnectionSelectionBlocker(run: HeartbeatRun, options: RunFailureRe
run.executionStage !== "preparing" || options.phase !== "setup" ||
run.exitCode != null || run.signal != null
) return false;
const configuration = asRecord(run.resultJson?.configurationIncomplete);
const recovery = asRecord(run.resultJson?.executionRecovery);
return configuration?.reason === "ai_connection_unavailable" &&
isAiConnectionConfigurationReason(configuration.selectionFailure) &&
return isOwnedAiConnectionSelectionRejection(run.resultJson?.configurationIncomplete) &&
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
}