From 6ba46fd0802ec0fbeee0172ea75e3ead4e4db935 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 15:06:29 -0600 Subject: [PATCH] fix(github): read only the webhook secrets each delivery step uses (#21651) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents receive GitHub work through a GitHub App chat channel. GitHub sends each event as a signed webhook delivery. > - Each delivery passes three steps: durable staging, the durable replay worker, and the replayed request's preflight under the credential lease. > - Each step resolved every stored GitHub App secret. A GitHub App connection stores seven secrets, so each delivery decrypted and audited about 21 secrets. The steps use only the signing secret, the bound installation id and the App id. > - On our staging instance, these reads are now the largest source of secret reads and `secret_access_events` rows. > - This pull request makes each webhook step read only its own keys, and only when it needs them. > - The benefit is fewer vault reads and audit inserts per delivery, and the App private key and OAuth client secret stay sealed on the webhook path. ## Linked Issues or Issue Description Refs #21641 (fixed the other source: review-check polls now reuse cached installation tokens). **What happened?** On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same window, `secret_access_events` recorded 97 resolutions of `credentials.webhookSecret`. It recorded the same count for `privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and `slug`. That is about 3 full credential resolutions per delivery, and about 21 audit rows per delivery (roughly 70–80 rows per minute). In the same window, 84 review-check polls caused no secret reads, because #21641 fixed that source. **What did you expect?** A delivery reads only the secrets it uses. An unsigned or forged delivery reads only the signing secret. **Steps to reproduce** 1. Configure a GitHub App chat endpoint. 2. Send one signed `issue_comment` delivery for the bound installation. 3. Count the new `secret_access_events` rows for the connection. Before this change, each stored key has 3 rows. A bad signature still reads every key once. The three call sites per delivery: 1. `server/src/services/chat-channels/github/webhook-ingress.ts` `stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs). 2. `server/src/services/chat-channels/github/webhook-ingress.ts` `processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to re-sign the retained body). This was the unattributed third resolution. 3. `server/src/services/chat-channels/github/webhook-authentication.ts` preflight and ping (`resolveCredentials` with no key filter). ## What Changed - Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook ingress module. It selects the secret refs for named keys. - Staging reads `webhookSecret` first. After the signature holds, it reads `installationId` (only when the event names an installation and is not an `installation` event) and `appId` (it scopes the recovery-receipt denial query). - The replay worker reads `webhookSecret` to re-sign the body. It reads `appId`, `privateKey` and `installationId` only for a recovered delivery that needs a source recheck. - The preflight and ping read `webhookSecret` through the existing `resolveCredentials` key filter. The preflight reads `installationId` only after the signature holds and only for an event that names an installation. - The failed-delivery recovery scan reads only `appId`, `privateKey` and `installationId`. - Unchanged: the timing-safe compare, the credential-mutation lease, the re-read of current endpoint state under the lease, the status and fence checks, and the superseded-credential retry. No cache of decrypted values. - Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to 6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad signature goes from all keys to 1 read. - `webhook-ingress.ts` was already above 1,000 lines. It grows by about 30 lines because the key selection belongs with the steps that use it. `chat-channels.integration.test.ts` holds the GitHub webhook fixtures, so the new test goes beside the existing signature tests. ## Verification - Red on master, green with this change: `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t "resolves only the secrets a"`. - `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3, privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3, installationId: 2, appId: 1}`. - `invalid_signature`: before, every key once. After, `{webhookSecret: 1}`. - Regression: GitHub webhook, signature, rotation, ping, recovery, installation, paused and revoked tests in `chat-channels.integration.test.ts` (filter `-t "[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack test (`materializes direct external-chat finals once…`, matched by the `ping` filter in "bookkeeping"). It passes when run alone, so it is a load flake and not related to this change. - `pnpm exec vitest run server/src/services/chat-channels/github/webhook-ingress.test.ts server/src/services/chat-github-provider-stress.test.ts server/src/services/chat-github-webhook-config.test.ts`: 105 passed. - `tsc --noEmit -p server/tsconfig.json`: passed. - `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the chat integration file pulls in a wide dependency set). I did not run that broad set locally. Required CI supplies that coverage for the final revision. ## Risks - Low risk. Each step reads a subset of the same refs from the same fenced record, so the values it uses do not change. - A missing `webhookSecret` ref still fails the signature check as before. A missing `installationId` ref still makes an installation-bearing event "ignored" as before. - The preflight now resolves `webhookSecret` and `installationId` in two reads under the same credential lease. The lease blocks credential mutation between them. - Installation lifecycle events (`installation-lifecycle.ts`) still resolve the full set, because they re-verify and re-persist all App credentials. They are rare. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code with tool use (shell, file edits, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../chat-channels.integration.test.ts | 39 +++++++++ .../github/webhook-authentication.ts | 14 ++- .../chat-channels/github/webhook-ingress.ts | 87 ++++++++++++++----- 3 files changed, 112 insertions(+), 28 deletions(-) diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index ff6ce27cc87..fe10f773b46 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -97,6 +97,7 @@ import { companyObservability } from "@paperclipai/db"; import * as runIdentityModule from "../services/run-identity.js"; import * as heartbeatModule from "../services/heartbeat.js"; import { chatGitHubRegistrations, chatGitHubReviews, chatGitHubConfigurations, toolCatalogEntries } from "@paperclipai/db"; +import { secretAccessEvents } from "@paperclipai/db"; import { AsyncLocalStorage } from "node:async_hooks"; import * as cloudRuntimeIdentity from "../services/cloud-runtime-identity.js"; import { @@ -6543,6 +6544,44 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { } } }); + it.each(["current", "invalid_signature"] as const)("resolves only the secrets a %s signed GitHub webhook delivery uses", async (kind) => { + const f = await reviewBotFixture(); + const [endpoint] = await readChatEndpointsById(f.endpoint.id); + const accessesBefore = new Set((await db.select({ id: secretAccessEvents.id }).from(secretAccessEvents) + .where(eq(secretAccessEvents.consumerId, endpoint.connectionId))).map(row => row.id)); + const delivery = randomUUID(); + const response = await f.service.handleWebhook(endpoint.publicId, "github", signedGitHubWebhookRequest({ + event: "issue_comment", delivery, webhookSecret: kind === "invalid_signature" ? "wrong-secret" : f.webhookSecret, + payload: { action: "created", installation: { id: 2468 }, + repository: { id: 97531, full_name: "paperclipai/paperclip", owner: { id: 1357, login: "paperclipai" } }, + issue: { number: 418 }, comment: { id: 41801, body: "Ordinary comment", user: { id: 42, login: "octocat" } }, sender: { id: 42, login: "octocat" } }, + })); + if (kind === "invalid_signature") expect(response.status).toBe(401); + else { + expect(response.status).toBeLessThan(300); + await expect.poll(async () => (await db.select({ status: chatActions.status }).from(chatActions).where(and( + eq(chatActions.endpointId, endpoint.id), eq(chatActions.providerActionId, `github_webhook_ingress:${delivery}`))))[0]?.status, + { timeout: 10_000 }).toBe("processed"); + } + const accesses = (await db.select({ id: secretAccessEvents.id, configPath: secretAccessEvents.configPath }).from(secretAccessEvents) + .where(eq(secretAccessEvents.consumerId, endpoint.connectionId))).filter(row => !accessesBefore.has(row.id)); + const reads: Record = {}; + for (const access of accesses) reads[access.configPath ?? "unknown"] = (reads[access.configPath ?? "unknown"] ?? 0) + 1; + // A delivery must never decrypt the App private key, OAuth client + // secret or other unused keys. Each step reads only what it uses: + // - durable staging verifies the provider signature (webhookSecret); + // once it holds, it rejects foreign installations (installationId) + // and scopes recovery-receipt denial to this App (appId); + // - the durable worker signs the retained body for internal replay + // (webhookSecret); + // - the replayed request re-authenticates under the credential lease + // against the current secret and installation (webhookSecret, + // installationId). + // A bad signature stops after the one webhookSecret read. + expect(reads).toEqual(kind === "invalid_signature" + ? { "credentials.webhookSecret": 1 } + : { "credentials.webhookSecret": 3, "credentials.installationId": 2, "credentials.appId": 1 }); + }); it("stack rejects forged raw gateway deliveries without changing verification or waking an agent", async () => { const f = await reviewBotFixture(); await db diff --git a/server/src/services/chat-channels/github/webhook-authentication.ts b/server/src/services/chat-channels/github/webhook-authentication.ts index d393306d084..ec94f94f416 100644 --- a/server/src/services/chat-channels/github/webhook-authentication.ts +++ b/server/src/services/chat-channels/github/webhook-authentication.ts @@ -117,7 +117,7 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio ) ) return false; - const credentials = await resolveCredentials(current.endpoint); + const credentials = await resolveCredentials(current.endpoint, undefined, ["webhookSecret"]); const expected = `sha256=${createHmac("sha256", credentials.webhookSecret).update(body).digest("hex")}`; let valid = false; try { @@ -182,7 +182,9 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio ) { return "ignored"; } - const credentials = await resolveCredentials(current.endpoint); + // Read only the signing secret until the signature holds; the bound + // installation is read below only for events that name one. + const credentials = await resolveCredentials(current.endpoint, undefined, ["webhookSecret"]); const expected = `sha256=${createHmac("sha256", credentials.webhookSecret).update(body).digest("hex")}`; let signatureValid = false; try { @@ -209,9 +211,13 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio const lifecycleEvent = ["github_app_authorization", "installation", "installation_repositories"].includes(eventType ?? ""); if (payload) { const incomingInstallationId = payload.installation?.id; + const boundInstallationId = + incomingInstallationId !== undefined && eventType !== "installation" + ? (await resolveCredentials(current.endpoint, undefined, ["installationId"])).installationId + : undefined; if ( incomingInstallationId !== undefined && - String(incomingInstallationId) !== credentials.installationId && + String(incomingInstallationId) !== boundInstallationId && eventType !== "installation" ) { // A dedicated endpoint represents exactly one GitHub App @@ -231,7 +237,7 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio if (!current.endpoint.setup.webhookVerifiedAt && ["active", "verifying", "attention"].includes(current.endpoint.status) && incomingInstallationId !== undefined && - String(incomingInstallationId) === credentials.installationId && + String(incomingInstallationId) === boundInstallationId && !lifecycleEvent) { await recordVerifiedWebhook(current.endpoint, credentialLease, inspection.headers.get("x-github-delivery")); } diff --git a/server/src/services/chat-channels/github/webhook-ingress.ts b/server/src/services/chat-channels/github/webhook-ingress.ts index e186b468a9a..1252528c7c6 100644 --- a/server/src/services/chat-channels/github/webhook-ingress.ts +++ b/server/src/services/chat-channels/github/webhook-ingress.ts @@ -133,6 +133,22 @@ export function githubWebhookIngressPayload( return payload as GitHubWebhookIngressPayload; } +/** + * Every resolved secret is a vault read plus an access-audit row, and a GitHub + * App connection stores seven of them. Webhook steps read only the keys they + * use; the App private key and OAuth client secret stay sealed on this path. + */ +export function githubCredentialRefsFor( + refs: ToolCredentialSecretRef[], + keys: readonly string[], +): ToolCredentialSecretRef[] { + return refs.filter((ref) => + keys.includes(ref.configPath.replace(/^credentials\./, "")), + ); +} + +const GITHUB_APP_TOKEN_KEYS = ["appId", "privateKey", "installationId"] as const; + export function redactedGitHubWebhookIngressPayload( payload: GitHubWebhookIngressPayload, ): RedactedGitHubWebhookIngressPayload { @@ -258,34 +274,31 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress return { kind: "ignored" }; } const context = runtimeContextForRecord(record); - let credentials: Record; - try { - credentials = await resolveCredentialRefs( - record.endpoint, - record.credentialSecretRefs, - ); - } catch (error) { + const credentialsSuperseded = async () => { const latest = await endpointRecord(endpoint.id); - if ( - latest && + return ( + latest !== null && (runtimeGeneration(latest.endpoint.setup) !== context.generation || credentialFingerprint(latest.credentialSecretRefs) !== context.credentialFingerprint) - ) { - continue; - } + ); + }; + const resolveKeys = (keys: readonly string[]) => + resolveCredentialRefs( + record.endpoint, + githubCredentialRefsFor(record.credentialSecretRefs, keys), + ); + let credentials: Record; + try { + // Unsigned or forged traffic must not decrypt anything beyond the + // signing secret. + credentials = await resolveKeys(["webhookSecret"]); + } catch (error) { + if (await credentialsSuperseded()) continue; throw error; } if (!githubSignatureIsValid(body, signature, credentials.webhookSecret)) { - const latest = await endpointRecord(endpoint.id); - if ( - latest && - (runtimeGeneration(latest.endpoint.setup) !== context.generation || - credentialFingerprint(latest.credentialSecretRefs) !== - context.credentialFingerprint) - ) { - continue; - } + if (await credentialsSuperseded()) continue; return { kind: "invalid_signature" }; } @@ -296,6 +309,21 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress ? (parsedPayload as { installation?: { id?: unknown } }).installation ?.id : undefined; + try { + // A signed event that names an installation is checked against the + // bound one; the App id scopes the recovery-receipt denial below. + Object.assign( + credentials, + await resolveKeys( + incomingInstallationId !== undefined && eventType !== "installation" + ? ["installationId", "appId"] + : ["appId"], + ), + ); + } catch (error) { + if (await credentialsSuperseded()) continue; + throw error; + } if ( incomingInstallationId !== undefined && String(incomingInstallationId) !== credentials.installationId && @@ -817,9 +845,11 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress } try { + // Replay re-signs the retained body for internal authentication. Only a + // recovered delivery also needs an App token to recheck its source. const credentials = await resolveCredentialRefs( record.endpoint, - record.credentialSecretRefs, + githubCredentialRefsFor(record.credentialSecretRefs, ["webhookSecret"]), ); const recovery = await db .select({ payload: chatActions.payload, status: chatActions.status }) @@ -840,7 +870,13 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress (recovery.status === "cancelled" || !(await githubRecoverySourceIsCurrent( (recovery.payload as GitHubRecoveryReceipt).original, - credentials, + await resolveCredentialRefs( + record.endpoint, + githubCredentialRefsFor( + record.credentialSecretRefs, + GITHUB_APP_TOKEN_KEYS, + ), + ), ))) ) { await settleGitHubWebhookIngress(action.id, { @@ -1217,7 +1253,10 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress } const credentials = await resolveCredentialRefs( record.endpoint, - record.credentialSecretRefs, + githubCredentialRefsFor( + record.credentialSecretRefs, + GITHUB_APP_TOKEN_KEYS, + ), ); if (credentials.appId !== window.appId || !credentials.installationId) return;