mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.
## Linked Issues or Issue Description
Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).
**What happened?**
On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z
to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same
window, `secret_access_events` recorded 97 resolutions of
`credentials.webhookSecret`. It recorded the same count for
`privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and
`slug`. That is about 3 full credential resolutions per delivery, and
about 21 audit rows per delivery (roughly 70–80 rows per minute). In the
same window, 84 review-check polls caused no secret reads, because
#21641 fixed that source.
**What did you expect?**
A delivery reads only the secrets it uses. An unsigned or forged
delivery reads only the signing secret.
**Steps to reproduce**
1. Configure a GitHub App chat endpoint.
2. Send one signed `issue_comment` delivery for the bound installation.
3. Count the new `secret_access_events` rows for the connection. Before
this change, each stored key has 3 rows. A bad signature still reads
every key once.
The three call sites per delivery:
1. `server/src/services/chat-channels/github/webhook-ingress.ts`
`stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs).
2. `server/src/services/chat-channels/github/webhook-ingress.ts`
`processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to
re-sign the retained body). This was the unattributed third resolution.
3. `server/src/services/chat-channels/github/webhook-authentication.ts`
preflight and ping (`resolveCredentials` with no key filter).
## What Changed
- Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook
ingress module. It selects the secret refs for named keys.
- Staging reads `webhookSecret` first. After the signature holds, it
reads `installationId` (only when the event names an installation and is
not an `installation` event) and `appId` (it scopes the recovery-receipt
denial query).
- The replay worker reads `webhookSecret` to re-sign the body. It reads
`appId`, `privateKey` and `installationId` only for a recovered delivery
that needs a source recheck.
- The preflight and ping read `webhookSecret` through the existing
`resolveCredentials` key filter. The preflight reads `installationId`
only after the signature holds and only for an event that names an
installation.
- The failed-delivery recovery scan reads only `appId`, `privateKey` and
`installationId`.
- Unchanged: the timing-safe compare, the credential-mutation lease, the
re-read of current endpoint state under the lease, the status and fence
checks, and the superseded-credential retry. No cache of decrypted
values.
- Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to
6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad
signature goes from all keys to 1 read.
- `webhook-ingress.ts` was already above 1,000 lines. It grows by about
30 lines because the key selection belongs with the steps that use it.
`chat-channels.integration.test.ts` holds the GitHub webhook fixtures,
so the new test goes beside the existing signature tests.
## Verification
- Red on master, green with this change: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t "resolves only
the secrets a"`.
- `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3,
privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3,
installationId: 2, appId: 1}`.
- `invalid_signature`: before, every key once. After, `{webhookSecret:
1}`.
- Regression: GitHub webhook, signature, rotation, ping, recovery,
installation, paused and revoked tests in
`chat-channels.integration.test.ts` (filter `-t
"[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub
callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack
test (`materializes direct external-chat finals once…`, matched by the
`ping` filter in "bookkeeping"). It passes when run alone, so it is a
load flake and not related to this change.
- `pnpm exec vitest run
server/src/services/chat-channels/github/webhook-ingress.test.ts
server/src/services/chat-github-provider-stress.test.ts
server/src/services/chat-github-webhook-config.test.ts`: 105 passed.
- `tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the
chat integration file pulls in a wide dependency set). I did not run
that broad set locally. Required CI supplies that coverage for the final
revision.
## Risks
- Low risk. Each step reads a subset of the same refs from the same
fenced record, so the values it uses do not change.
- A missing `webhookSecret` ref still fails the signature check as
before. A missing `installationId` ref still makes an
installation-bearing event "ignored" as before.
- The preflight now resolves `webhookSecret` and `installationId` in two
reads under the same credential lease. The lease blocks credential
mutation between them.
- Installation lifecycle events (`installation-lifecycle.ts`) still
resolve the full set, because they re-verify and re-persist all App
credentials. They are rare.
- No migration and no API change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution) and extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ddfbbfff20
commit
6ba46fd080
3 files changed
+112
-28
No files matched your search
@@ -97,6 +97,7 @@ import { companyObservability } from "@paperclipai/db";
|
||||
import * as runIdentityModule from "../services/run-identity.js";
|
||||
import * as heartbeatModule from "../services/heartbeat.js";
|
||||
import { chatGitHubRegistrations, chatGitHubReviews, chatGitHubConfigurations, toolCatalogEntries } from "@paperclipai/db";
|
||||
import { secretAccessEvents } from "@paperclipai/db";
|
||||
import { AsyncLocalStorage } from "node:async_hooks";
|
||||
import * as cloudRuntimeIdentity from "../services/cloud-runtime-identity.js";
|
||||
import {
|
||||
@@ -6543,6 +6544,44 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
}
|
||||
});
|
||||
it.each(["current", "invalid_signature"] as const)("resolves only the secrets a %s signed GitHub webhook delivery uses", async (kind) => {
|
||||
const f = await reviewBotFixture();
|
||||
const [endpoint] = await readChatEndpointsById(f.endpoint.id);
|
||||
const accessesBefore = new Set((await db.select({ id: secretAccessEvents.id }).from(secretAccessEvents)
|
||||
.where(eq(secretAccessEvents.consumerId, endpoint.connectionId))).map(row => row.id));
|
||||
const delivery = randomUUID();
|
||||
const response = await f.service.handleWebhook(endpoint.publicId, "github", signedGitHubWebhookRequest({
|
||||
event: "issue_comment", delivery, webhookSecret: kind === "invalid_signature" ? "wrong-secret" : f.webhookSecret,
|
||||
payload: { action: "created", installation: { id: 2468 },
|
||||
repository: { id: 97531, full_name: "paperclipai/paperclip", owner: { id: 1357, login: "paperclipai" } },
|
||||
issue: { number: 418 }, comment: { id: 41801, body: "Ordinary comment", user: { id: 42, login: "octocat" } }, sender: { id: 42, login: "octocat" } },
|
||||
}));
|
||||
if (kind === "invalid_signature") expect(response.status).toBe(401);
|
||||
else {
|
||||
expect(response.status).toBeLessThan(300);
|
||||
await expect.poll(async () => (await db.select({ status: chatActions.status }).from(chatActions).where(and(
|
||||
eq(chatActions.endpointId, endpoint.id), eq(chatActions.providerActionId, `github_webhook_ingress:${delivery}`))))[0]?.status,
|
||||
{ timeout: 10_000 }).toBe("processed");
|
||||
}
|
||||
const accesses = (await db.select({ id: secretAccessEvents.id, configPath: secretAccessEvents.configPath }).from(secretAccessEvents)
|
||||
.where(eq(secretAccessEvents.consumerId, endpoint.connectionId))).filter(row => !accessesBefore.has(row.id));
|
||||
const reads: Record<string, number> = {};
|
||||
for (const access of accesses) reads[access.configPath ?? "unknown"] = (reads[access.configPath ?? "unknown"] ?? 0) + 1;
|
||||
// A delivery must never decrypt the App private key, OAuth client
|
||||
// secret or other unused keys. Each step reads only what it uses:
|
||||
// - durable staging verifies the provider signature (webhookSecret);
|
||||
// once it holds, it rejects foreign installations (installationId)
|
||||
// and scopes recovery-receipt denial to this App (appId);
|
||||
// - the durable worker signs the retained body for internal replay
|
||||
// (webhookSecret);
|
||||
// - the replayed request re-authenticates under the credential lease
|
||||
// against the current secret and installation (webhookSecret,
|
||||
// installationId).
|
||||
// A bad signature stops after the one webhookSecret read.
|
||||
expect(reads).toEqual(kind === "invalid_signature"
|
||||
? { "credentials.webhookSecret": 1 }
|
||||
: { "credentials.webhookSecret": 3, "credentials.installationId": 2, "credentials.appId": 1 });
|
||||
});
|
||||
it("stack rejects forged raw gateway deliveries without changing verification or waking an agent", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
await db
|
||||
|
||||
@@ -117,7 +117,7 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio
|
||||
)
|
||||
)
|
||||
return false;
|
||||
const credentials = await resolveCredentials(current.endpoint);
|
||||
const credentials = await resolveCredentials(current.endpoint, undefined, ["webhookSecret"]);
|
||||
const expected = `sha256=${createHmac("sha256", credentials.webhookSecret).update(body).digest("hex")}`;
|
||||
let valid = false;
|
||||
try {
|
||||
@@ -182,7 +182,9 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio
|
||||
) {
|
||||
return "ignored";
|
||||
}
|
||||
const credentials = await resolveCredentials(current.endpoint);
|
||||
// Read only the signing secret until the signature holds; the bound
|
||||
// installation is read below only for events that name one.
|
||||
const credentials = await resolveCredentials(current.endpoint, undefined, ["webhookSecret"]);
|
||||
const expected = `sha256=${createHmac("sha256", credentials.webhookSecret).update(body).digest("hex")}`;
|
||||
let signatureValid = false;
|
||||
try {
|
||||
@@ -209,9 +211,13 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio
|
||||
const lifecycleEvent = ["github_app_authorization", "installation", "installation_repositories"].includes(eventType ?? "");
|
||||
if (payload) {
|
||||
const incomingInstallationId = payload.installation?.id;
|
||||
const boundInstallationId =
|
||||
incomingInstallationId !== undefined && eventType !== "installation"
|
||||
? (await resolveCredentials(current.endpoint, undefined, ["installationId"])).installationId
|
||||
: undefined;
|
||||
if (
|
||||
incomingInstallationId !== undefined &&
|
||||
String(incomingInstallationId) !== credentials.installationId &&
|
||||
String(incomingInstallationId) !== boundInstallationId &&
|
||||
eventType !== "installation"
|
||||
) {
|
||||
// A dedicated endpoint represents exactly one GitHub App
|
||||
@@ -231,7 +237,7 @@ export function createGitHubWebhookAuthentication(db: Db, options: Authenticatio
|
||||
if (!current.endpoint.setup.webhookVerifiedAt &&
|
||||
["active", "verifying", "attention"].includes(current.endpoint.status) &&
|
||||
incomingInstallationId !== undefined &&
|
||||
String(incomingInstallationId) === credentials.installationId &&
|
||||
String(incomingInstallationId) === boundInstallationId &&
|
||||
!lifecycleEvent) {
|
||||
await recordVerifiedWebhook(current.endpoint, credentialLease, inspection.headers.get("x-github-delivery"));
|
||||
}
|
||||
|
||||
@@ -133,6 +133,22 @@ export function githubWebhookIngressPayload(
|
||||
return payload as GitHubWebhookIngressPayload;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every resolved secret is a vault read plus an access-audit row, and a GitHub
|
||||
* App connection stores seven of them. Webhook steps read only the keys they
|
||||
* use; the App private key and OAuth client secret stay sealed on this path.
|
||||
*/
|
||||
export function githubCredentialRefsFor(
|
||||
refs: ToolCredentialSecretRef[],
|
||||
keys: readonly string[],
|
||||
): ToolCredentialSecretRef[] {
|
||||
return refs.filter((ref) =>
|
||||
keys.includes(ref.configPath.replace(/^credentials\./, "")),
|
||||
);
|
||||
}
|
||||
|
||||
const GITHUB_APP_TOKEN_KEYS = ["appId", "privateKey", "installationId"] as const;
|
||||
|
||||
export function redactedGitHubWebhookIngressPayload(
|
||||
payload: GitHubWebhookIngressPayload,
|
||||
): RedactedGitHubWebhookIngressPayload {
|
||||
@@ -258,34 +274,31 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress
|
||||
return { kind: "ignored" };
|
||||
}
|
||||
const context = runtimeContextForRecord(record);
|
||||
let credentials: Record<string, string>;
|
||||
try {
|
||||
credentials = await resolveCredentialRefs(
|
||||
record.endpoint,
|
||||
record.credentialSecretRefs,
|
||||
);
|
||||
} catch (error) {
|
||||
const credentialsSuperseded = async () => {
|
||||
const latest = await endpointRecord(endpoint.id);
|
||||
if (
|
||||
latest &&
|
||||
return (
|
||||
latest !== null &&
|
||||
(runtimeGeneration(latest.endpoint.setup) !== context.generation ||
|
||||
credentialFingerprint(latest.credentialSecretRefs) !==
|
||||
context.credentialFingerprint)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
);
|
||||
};
|
||||
const resolveKeys = (keys: readonly string[]) =>
|
||||
resolveCredentialRefs(
|
||||
record.endpoint,
|
||||
githubCredentialRefsFor(record.credentialSecretRefs, keys),
|
||||
);
|
||||
let credentials: Record<string, string>;
|
||||
try {
|
||||
// Unsigned or forged traffic must not decrypt anything beyond the
|
||||
// signing secret.
|
||||
credentials = await resolveKeys(["webhookSecret"]);
|
||||
} catch (error) {
|
||||
if (await credentialsSuperseded()) continue;
|
||||
throw error;
|
||||
}
|
||||
if (!githubSignatureIsValid(body, signature, credentials.webhookSecret)) {
|
||||
const latest = await endpointRecord(endpoint.id);
|
||||
if (
|
||||
latest &&
|
||||
(runtimeGeneration(latest.endpoint.setup) !== context.generation ||
|
||||
credentialFingerprint(latest.credentialSecretRefs) !==
|
||||
context.credentialFingerprint)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
if (await credentialsSuperseded()) continue;
|
||||
return { kind: "invalid_signature" };
|
||||
}
|
||||
|
||||
@@ -296,6 +309,21 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress
|
||||
? (parsedPayload as { installation?: { id?: unknown } }).installation
|
||||
?.id
|
||||
: undefined;
|
||||
try {
|
||||
// A signed event that names an installation is checked against the
|
||||
// bound one; the App id scopes the recovery-receipt denial below.
|
||||
Object.assign(
|
||||
credentials,
|
||||
await resolveKeys(
|
||||
incomingInstallationId !== undefined && eventType !== "installation"
|
||||
? ["installationId", "appId"]
|
||||
: ["appId"],
|
||||
),
|
||||
);
|
||||
} catch (error) {
|
||||
if (await credentialsSuperseded()) continue;
|
||||
throw error;
|
||||
}
|
||||
if (
|
||||
incomingInstallationId !== undefined &&
|
||||
String(incomingInstallationId) !== credentials.installationId &&
|
||||
@@ -817,9 +845,11 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress
|
||||
}
|
||||
|
||||
try {
|
||||
// Replay re-signs the retained body for internal authentication. Only a
|
||||
// recovered delivery also needs an App token to recheck its source.
|
||||
const credentials = await resolveCredentialRefs(
|
||||
record.endpoint,
|
||||
record.credentialSecretRefs,
|
||||
githubCredentialRefsFor(record.credentialSecretRefs, ["webhookSecret"]),
|
||||
);
|
||||
const recovery = await db
|
||||
.select({ payload: chatActions.payload, status: chatActions.status })
|
||||
@@ -840,7 +870,13 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress
|
||||
(recovery.status === "cancelled" ||
|
||||
!(await githubRecoverySourceIsCurrent(
|
||||
(recovery.payload as GitHubRecoveryReceipt).original,
|
||||
credentials,
|
||||
await resolveCredentialRefs(
|
||||
record.endpoint,
|
||||
githubCredentialRefsFor(
|
||||
record.credentialSecretRefs,
|
||||
GITHUB_APP_TOKEN_KEYS,
|
||||
),
|
||||
),
|
||||
)))
|
||||
) {
|
||||
await settleGitHubWebhookIngress(action.id, {
|
||||
@@ -1217,7 +1253,10 @@ export function createGitHubWebhookIngress(db: Db, options: GitHubWebhookIngress
|
||||
}
|
||||
const credentials = await resolveCredentialRefs(
|
||||
record.endpoint,
|
||||
record.credentialSecretRefs,
|
||||
githubCredentialRefsFor(
|
||||
record.credentialSecretRefs,
|
||||
GITHUB_APP_TOKEN_KEYS,
|
||||
),
|
||||
);
|
||||
if (credentials.appId !== window.appId || !credentials.installationId)
|
||||
return;
|
||||
|
||||
Reference in new issue
Block a user