mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel lets an agent review pull requests and answer comments through a GitHub App > - Each GitHub call gets a repository-scoped installation token from `githubBotRepositoryToken` > - That function minted a new token on every call. To mint, it resolved every App secret through the secret service, and each resolution wrote one audit row > - The review-check head poll calls it about every 5 minutes for each waiting check. On staging this made the secret audit insert one of the busiest statements in the database > - This pull request keeps a minted token in memory until shortly before it expires, and checks the connection's current state on every call > - The benefit is far fewer database writes and reads, fewer GitHub token mints, and an audit log that records real secret use ## Linked Issues or Issue Description **What happened?** On paperclip.staging (commita18479bbba, measured at 19:05Z on 2026-11-14), `secret_access_events` got about 285 rows in 60 s. These were 44 resolutions each of the GitHub App connection's `credentials.appId`, `privateKey`, `installationId`, `webhookSecret`, `clientId` and `clientSecret` (+21 `slug`), with `consumer_type` `tool_connection` and actor `system`. That is about 4.7 audit inserts/s. The matching `secret_versions` and binding reads ran at about 3.6/s each in Neon. These were among the busiest statements on the database. **Expected behavior** A GitHub installation token is valid for one hour. Paperclip should reuse it and resolve the App secrets only when it mints a new token. **Steps to reproduce** 1. Connect a GitHub App chat endpoint and get pull request reviews into the `received` review-check state (staging had 84 `github_review_check` actions). 2. Let the review-check worker poll the pull request head (23 polls/min measured on staging). 3. Count `secret_access_events` rows per minute. Each poll adds one row per App secret and mints a new token at GitHub. **Paperclip version or commit** master ata18479bbba(the commit serving paperclip.staging at the time of measurement). **Where in the code** `githubBotRepositoryToken` in `server/src/services/chat-github-client.ts`. Callers: `chat-github-checks.ts` (review-check head poll), `chat-github-response-comments.ts` and `chat-github-reviews.ts`. ## What Changed - `githubBotRepositoryToken` keeps a minted token in a per-process, in-memory cache (at most 512 entries, least recently used first out). It never persists or logs tokens. - The cache uses the token until 5 minutes before GitHub's `expires_at` (capped at one hour). If GitHub does not give a usable expiry, the token is not kept. - The cache key holds the company, endpoint, repository id, the exact permission set, and an authority revision. The revision holds the connection id, the endpoint's runtime generation (pause, revoke and availability changes advance it), and for each mint credential its secret ref, version selector, secret status, current version, `updatedAt`, and binding. The installation id is a secret, so its secret ref and version identify it in the key. - Each call reads the current authority from the database in two narrow statements: one row from the endpoint and connection join (status, enabled, refs, runtime generation), and the three mint secrets with their bindings (status and version columns only, no secret values). The endpoint and connection status checks run on every call, as before. - If a status check fails, all cached tokens for that endpoint are dropped. A token cannot survive a revoke, pause, disable or credential change. - Secrets are resolved only when a token is minted, and only the three that minting uses (`appId`, `privateKey`, `installationId`). So the audit log gets three rows per mint instead of one row per App secret per call. - New `githubBotRepositoryClient` sends repository requests. If GitHub returns 401 for a cached token, it drops that token, mints a new one, and sends the request once more. A 401 for a newly minted token is not retried. The review-check, response-comment and review paths use this client. - The token policy stays in the GitHub chat client module (`rules/module-boundaries.mdc`). The callers only change how they get a request function. ## Verification - New test file `server/src/__tests__/github-repository-token-cache.test.ts` (embedded Postgres, real secret service, mocked GitHub fetch): - `mints once and resolves the App's secrets only when it mints`: red on master (second call minted `installation-token-2`), green now. It also checks that only the three mint secrets are audited. - `refreshes a token shortly before GitHub's expiry`: reused at +54 min, minted again at +56 min. - `never keeps a token GitHub issues for less than the safety margin` - `mints again after a credential rotation and uses the new credential`: rotates the installation id secret and checks that the new mint uses it. - `mints again after the connection's credential refs change` - `never serves a cached token across a pause, revoke or disabled connection`: includes a pause and resume with no call in between. - `mints a separate token for each repository` - `replaces a cached token GitHub rejects and retries the request once`: also checks that a 401 for a new token is not retried. - On master, 5 of the 8 tests fail. All 8 pass with this change. - Focused suites pass locally: every server test file that matches `github`, plus `chat-sdk-runtime.adapters`, `chat-provider-inventory`, `idle-completed-history` and `connection-management.integration` (41 of 42 files, 955 of 956 tests). The one failure, `github-connection-enrollment-work.test.ts`, also fails on unchanged master (`expected +0 to be 1`), so this change did not cause it. - `pnpm --filter @paperclipai/server typecheck` passes. - CI supplies the full typecheck, build and test coverage for the final revision. ## Risks - Low risk. A cached token is used only while the endpoint, connection and mint secrets are unchanged, and the status checks still read current database state on every call. - A GitHub-side revoke of a token (for example, an App uninstall) makes GitHub return 401. The client then mints again once. If the installation is gone, the mint fails as before. - The cache is per process. Each server process mints its own token, at most once per hour per repository and permission set. - Two concurrent first calls for the same key can both mint. This is the same as before and is rare. - No schema, migration or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a18479bbba
commit
163d9f323c
5 files changed
+522
-52
No files matched your search
@@ -0,0 +1,217 @@
|
||||
import { generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import {
|
||||
agents,
|
||||
chatEndpoints,
|
||||
companies,
|
||||
companySecretBindings,
|
||||
createDb,
|
||||
secretAccessEvents,
|
||||
toolApplications,
|
||||
toolConnections,
|
||||
} from "@paperclipai/db";
|
||||
import type { ToolCredentialSecretRef } from "@paperclipai/shared";
|
||||
import {
|
||||
githubBotRepositoryClient,
|
||||
githubBotRepositoryToken,
|
||||
} from "../services/chat-github-client.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const describeDatabase = support.supported ? describe : describe.skip;
|
||||
|
||||
const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });
|
||||
const privateKeyPem = privateKey.export({ type: "pkcs8", format: "pem" }).toString();
|
||||
const HOUR = 60 * 60_000;
|
||||
|
||||
describeDatabase("GitHub repository token reuse", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-github-token-cache-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 30_000);
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 0 });
|
||||
await database?.cleanup();
|
||||
});
|
||||
|
||||
async function fixture() {
|
||||
const companyId = randomUUID(), applicationId = randomUUID(), connectionId = randomUUID(), endpointId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "GitHub token reuse", issuePrefix: randomUUID().slice(0, 8).toUpperCase() });
|
||||
const agentId = randomUUID();
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "process" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: "GitHub", type: "chat" });
|
||||
const secrets = secretService(db);
|
||||
const values = { appId: "1234", privateKey: privateKeyPem, installationId: "2468", webhookSecret: "webhook-secret" };
|
||||
const refs: ToolCredentialSecretRef[] = [];
|
||||
const secretIds: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(values)) {
|
||||
const secret = await secrets.create(companyId, {
|
||||
name: `github.${key}.${randomUUID().slice(0, 8)}`,
|
||||
key: `CHAT_GITHUB_${key}_${randomUUID()}`.replaceAll("-", "_").toUpperCase(),
|
||||
provider: "local_encrypted",
|
||||
value,
|
||||
});
|
||||
secretIds[key] = secret.id;
|
||||
refs.push({ secretId: secret.id, versionSelector: "latest", configPath: `credentials.${key}`, required: true });
|
||||
}
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "GitHub", uid: connectionId,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active", credentialSecretRefs: refs });
|
||||
await db.insert(companySecretBindings).values(refs.map(ref => ({ companyId, secretId: ref.secretId,
|
||||
targetType: "tool_connection", targetId: connectionId, configPath: ref.configPath })));
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, provider: "github",
|
||||
publicId: endpointId, assignedAgentId: agentId, status: "active" });
|
||||
|
||||
const mints: Array<{ installationId: string; repositoryIds: number[]; permissions: unknown }> = [];
|
||||
const requests: Array<{ path: string; token: string }> = [];
|
||||
const rejected = new Set<string>();
|
||||
let expiresIn = HOUR;
|
||||
const fetchImpl = vi.fn(async (input: string | URL | Request, init?: RequestInit) => {
|
||||
const url = new URL(String(input));
|
||||
const token = String(new Headers(init?.headers).get("authorization") ?? "").replace(/^Bearer /, "");
|
||||
const mint = url.pathname.match(/^\/app\/installations\/(\d+)\/access_tokens$/);
|
||||
if (mint) {
|
||||
const body = JSON.parse(String(init?.body)) as { repository_ids: number[]; permissions: unknown };
|
||||
mints.push({ installationId: mint[1]!, repositoryIds: body.repository_ids, permissions: body.permissions });
|
||||
return Response.json({ token: `installation-token-${mints.length}`,
|
||||
expires_at: new Date(Date.now() + expiresIn).toISOString() }, { status: 201 });
|
||||
}
|
||||
requests.push({ path: url.pathname, token });
|
||||
if (rejected.has(token)) return Response.json({ message: "Bad credentials" }, { status: 401 });
|
||||
return Response.json({ head: { sha: "abc" } });
|
||||
}) as unknown as typeof fetch & ReturnType<typeof vi.fn>;
|
||||
const audits = async () => (await db.select({ configPath: secretAccessEvents.configPath }).from(secretAccessEvents)
|
||||
.where(and(eq(secretAccessEvents.companyId, companyId), eq(secretAccessEvents.consumerId, connectionId))))
|
||||
.map(row => row.configPath).sort();
|
||||
const token = (repositoryId = "101") => githubBotRepositoryToken(db, companyId, endpointId, repositoryId, fetchImpl);
|
||||
return { companyId, connectionId, endpointId, secretIds, mints, requests, rejected, fetchImpl, audits, token,
|
||||
setExpiresIn: (ms: number) => { expiresIn = ms; } };
|
||||
}
|
||||
|
||||
it("mints once and resolves the App's secrets only when it mints", async () => {
|
||||
const f = await fixture();
|
||||
const first = await f.token();
|
||||
const second = await f.token();
|
||||
expect(second).toBe(first);
|
||||
expect(f.mints).toEqual([{ installationId: "2468", repositoryIds: [101],
|
||||
permissions: { contents: "read", metadata: "read", issues: "write", pull_requests: "write", checks: "write" } }]);
|
||||
// One audited read per mint credential; the webhook secret is never read to mint.
|
||||
expect(await f.audits()).toEqual(["credentials.appId", "credentials.installationId", "credentials.privateKey"]);
|
||||
});
|
||||
|
||||
it("refreshes a token shortly before GitHub's expiry", async () => {
|
||||
const f = await fixture();
|
||||
const start = Date.now();
|
||||
const now = vi.spyOn(Date, "now").mockReturnValue(start);
|
||||
const first = await f.token();
|
||||
now.mockReturnValue(start + 54 * 60_000);
|
||||
expect(await f.token()).toBe(first);
|
||||
now.mockReturnValue(start + 56 * 60_000);
|
||||
const refreshed = await f.token();
|
||||
expect(refreshed).not.toBe(first);
|
||||
expect(f.mints).toHaveLength(2);
|
||||
expect(await f.token()).toBe(refreshed);
|
||||
});
|
||||
|
||||
it("never keeps a token GitHub issues for less than the safety margin", async () => {
|
||||
const f = await fixture();
|
||||
f.setExpiresIn(2 * 60_000);
|
||||
await f.token();
|
||||
await f.token();
|
||||
expect(f.mints).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("mints again after a credential rotation and uses the new credential", async () => {
|
||||
const f = await fixture();
|
||||
const first = await f.token();
|
||||
await secretService(db).rotate(f.secretIds.installationId!, { value: "1357" });
|
||||
const rotated = await f.token();
|
||||
expect(rotated).not.toBe(first);
|
||||
expect(f.mints.map(mint => mint.installationId)).toEqual(["2468", "1357"]);
|
||||
expect(await f.token()).toBe(rotated);
|
||||
expect(f.mints).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("mints again after the connection's credential refs change", async () => {
|
||||
const f = await fixture();
|
||||
const first = await f.token();
|
||||
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, f.connectionId));
|
||||
// Same secrets under a pinned version: a different authority, so a different token.
|
||||
await db.update(toolConnections).set({ credentialSecretRefs: connection!.credentialSecretRefs
|
||||
.map(ref => ({ ...ref, versionSelector: 1 })) }).where(eq(toolConnections.id, f.connectionId));
|
||||
expect(await f.token()).not.toBe(first);
|
||||
expect(f.mints).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("never serves a cached token across a pause, revoke or disabled connection", async () => {
|
||||
const f = await fixture();
|
||||
const endpoint = eq(chatEndpoints.id, f.endpointId);
|
||||
const first = await f.token();
|
||||
|
||||
await db.update(chatEndpoints).set({ status: "paused" }).where(endpoint);
|
||||
await expect(f.token()).rejects.toThrow("GitHub bot connection is unavailable");
|
||||
await db.update(chatEndpoints).set({ status: "active" }).where(endpoint);
|
||||
const afterPause = await f.token();
|
||||
expect(afterPause).not.toBe(first);
|
||||
|
||||
// A pause and resume with no call in between still drops the token: pausing
|
||||
// advances the endpoint's runtime generation.
|
||||
await db.update(chatEndpoints).set({ status: "paused",
|
||||
setup: sql`jsonb_set(${chatEndpoints.setup}, '{runtimeGeneration}', to_jsonb(coalesce((${chatEndpoints.setup}->>'runtimeGeneration')::int, 0) + 1))` })
|
||||
.where(endpoint);
|
||||
await db.update(chatEndpoints).set({ status: "active" }).where(endpoint);
|
||||
const afterResume = await f.token();
|
||||
expect(afterResume).not.toBe(afterPause);
|
||||
|
||||
await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connectionId));
|
||||
await expect(f.token()).rejects.toThrow("GitHub bot connection is not active");
|
||||
await db.update(toolConnections).set({ enabled: true }).where(eq(toolConnections.id, f.connectionId));
|
||||
const afterDisable = await f.token();
|
||||
expect(afterDisable).not.toBe(afterResume);
|
||||
|
||||
await db.update(chatEndpoints).set({ status: "revoked" }).where(endpoint);
|
||||
await expect(f.token()).rejects.toThrow("GitHub bot connection is unavailable");
|
||||
expect(f.mints).toHaveLength(4);
|
||||
});
|
||||
|
||||
it("mints a separate token for each repository", async () => {
|
||||
const f = await fixture();
|
||||
const first = await f.token("101");
|
||||
const second = await f.token("202");
|
||||
expect(second).not.toBe(first);
|
||||
expect(await f.token("101")).toBe(first);
|
||||
expect(await f.token("202")).toBe(second);
|
||||
expect(f.mints.map(mint => mint.repositoryIds)).toEqual([[101], [202]]);
|
||||
});
|
||||
|
||||
it("replaces a cached token GitHub rejects and retries the request once", async () => {
|
||||
const f = await fixture();
|
||||
const cached = await f.token();
|
||||
f.rejected.add(cached);
|
||||
const request = await githubBotRepositoryClient(db, f.companyId, f.endpointId, "101", f.fetchImpl);
|
||||
await expect(request<{ head: { sha: string } }>("/repos/o/r/pulls/1")).resolves.toEqual({ head: { sha: "abc" } });
|
||||
expect(f.requests).toEqual([
|
||||
{ path: "/repos/o/r/pulls/1", token: cached },
|
||||
{ path: "/repos/o/r/pulls/1", token: "installation-token-2" },
|
||||
]);
|
||||
// The replacement is the cached token from now on.
|
||||
expect(await f.token()).toBe("installation-token-2");
|
||||
expect(f.mints).toHaveLength(2);
|
||||
|
||||
// A freshly minted token that GitHub rejects is not retried.
|
||||
const other = await fixture();
|
||||
other.rejected.add("installation-token-1");
|
||||
const fresh = await githubBotRepositoryClient(db, other.companyId, other.endpointId, "101", other.fetchImpl);
|
||||
await expect(fresh("/repos/o/r/pulls/1")).rejects.toMatchObject({ details: { providerStatus: 401 } });
|
||||
expect(other.requests).toHaveLength(1);
|
||||
expect(other.mints).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -17,7 +17,7 @@ import {
|
||||
import type { GitHubReviewEventContext } from "@paperclipai/shared";
|
||||
import { runtimePublicOrigin } from "./cloud-runtime-identity.js";
|
||||
import {
|
||||
githubBotRepositoryToken,
|
||||
githubBotRepositoryClient,
|
||||
githubBotRequest,
|
||||
} from "./chat-github-client.js";
|
||||
|
||||
@@ -132,7 +132,7 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) {
|
||||
: state === "running"
|
||||
? "Agent is reviewing this commit"
|
||||
: "Waiting for the assigned agent";
|
||||
const token = await githubBotRepositoryToken(
|
||||
const github = await githubBotRepositoryClient(
|
||||
db,
|
||||
action.companyId,
|
||||
action.endpointId,
|
||||
@@ -143,13 +143,7 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) {
|
||||
const request = <T>(
|
||||
path: string,
|
||||
options?: Parameters<typeof githubBotRequest>[3],
|
||||
) =>
|
||||
githubBotRequest<T>(
|
||||
lease.fetch,
|
||||
token,
|
||||
`${prefix}${path}`,
|
||||
options,
|
||||
);
|
||||
) => github<T>(`${prefix}${path}`, options);
|
||||
const current = await request<{ head: { sha: string } }>(
|
||||
`/pulls/${event.pullNumber}`,
|
||||
);
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
import { createPrivateKey, createSign } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { chatEndpoints, toolConnections, type Db } from "@paperclipai/db";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import {
|
||||
chatEndpoints,
|
||||
companySecretBindings,
|
||||
companySecrets,
|
||||
toolConnections,
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { secretService } from "./secrets.js";
|
||||
import { conflict, forbidden, unprocessable } from "../errors.js";
|
||||
import { HttpError, conflict, forbidden, unprocessable } from "../errors.js";
|
||||
import { githubPublicationRetry, githubProviderRetry, GITHUB_INLINE_RETRY_WINDOW_MS, GITHUB_INLINE_MAX_ATTEMPTS } from "./chat-channels/github/retry.js";
|
||||
|
||||
export function githubAppJwt(
|
||||
@@ -200,7 +206,238 @@ export async function githubBotCredentials(
|
||||
};
|
||||
}
|
||||
|
||||
/** Least-privilege short-lived token restricted to the single task repository. */
|
||||
const GITHUB_REPOSITORY_TOKEN_PERMISSIONS = {
|
||||
contents: "read",
|
||||
metadata: "read",
|
||||
issues: "write",
|
||||
pull_requests: "write",
|
||||
checks: "write",
|
||||
} as const;
|
||||
/** Only these secrets are needed to mint; the App's other credentials are not read here. */
|
||||
const GITHUB_MINT_CREDENTIALS = ["appId", "privateKey", "installationId"] as const;
|
||||
const GITHUB_REPOSITORY_TOKEN_REFRESH_MARGIN_MS = 5 * 60_000;
|
||||
const GITHUB_REPOSITORY_TOKEN_MAX_LIFETIME_MS = 60 * 60_000;
|
||||
const GITHUB_REPOSITORY_TOKEN_CACHE_LIMIT = 512;
|
||||
|
||||
type CachedRepositoryToken = { token: string; endpointId: string; refreshAt: number };
|
||||
type RepositoryToken = { token: string; key: string; cached: boolean };
|
||||
|
||||
// Per process and in memory only. Tokens are never persisted or logged.
|
||||
const repositoryTokens = new Map<string, CachedRepositoryToken>();
|
||||
|
||||
function forgetEndpointRepositoryTokens(endpointId: string) {
|
||||
for (const [key, entry] of repositoryTokens)
|
||||
if (entry.endpointId === endpointId) repositoryTokens.delete(key);
|
||||
}
|
||||
|
||||
function rememberRepositoryToken(key: string, entry: CachedRepositoryToken) {
|
||||
repositoryTokens.delete(key);
|
||||
if (repositoryTokens.size >= GITHUB_REPOSITORY_TOKEN_CACHE_LIMIT) {
|
||||
const now = Date.now();
|
||||
for (const [stale, cached] of repositoryTokens)
|
||||
if (cached.refreshAt <= now) repositoryTokens.delete(stale);
|
||||
}
|
||||
// Map order is least recently used first.
|
||||
for (const oldest of repositoryTokens.keys()) {
|
||||
if (repositoryTokens.size < GITHUB_REPOSITORY_TOKEN_CACHE_LIMIT) break;
|
||||
repositoryTokens.delete(oldest);
|
||||
}
|
||||
repositoryTokens.set(key, entry);
|
||||
}
|
||||
|
||||
/**
|
||||
* Current minting authority, read from the database on every call. The
|
||||
* revision covers everything that changes what a token may do: the endpoint's
|
||||
* runtime generation (bumped on pause, revoke and availability changes), the
|
||||
* connection, and each mint credential's secret ref, selected version, state
|
||||
* and binding. Any change makes every token cached under the old revision
|
||||
* unreachable. Secret values are not read here.
|
||||
*/
|
||||
async function githubBotRepositoryAuthority(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
endpointId: string,
|
||||
) {
|
||||
const [row] = await db
|
||||
.select({
|
||||
endpointStatus: chatEndpoints.status,
|
||||
runtimeGeneration: sql<unknown>`${chatEndpoints.setup}->'runtimeGeneration'`,
|
||||
connectionId: toolConnections.id,
|
||||
enabled: toolConnections.enabled,
|
||||
connectionStatus: toolConnections.status,
|
||||
refs: toolConnections.credentialSecretRefs,
|
||||
})
|
||||
.from(chatEndpoints)
|
||||
.innerJoin(
|
||||
toolConnections,
|
||||
and(
|
||||
eq(toolConnections.id, chatEndpoints.connectionId),
|
||||
eq(toolConnections.companyId, companyId),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(chatEndpoints.companyId, companyId),
|
||||
eq(chatEndpoints.id, endpointId),
|
||||
eq(chatEndpoints.provider, "github"),
|
||||
),
|
||||
);
|
||||
if (!row || ["archived", "revoked", "paused"].includes(row.endpointStatus))
|
||||
throw conflict("GitHub bot connection is unavailable");
|
||||
const [appId, privateKey, installationId] = GITHUB_MINT_CREDENTIALS.map(
|
||||
(key) => row.refs.find((ref) => ref.configPath === `credentials.${key}`),
|
||||
);
|
||||
if (!appId || !privateKey)
|
||||
throw conflict("Connect the GitHub App before continuing");
|
||||
if (
|
||||
!row.enabled ||
|
||||
row.connectionStatus !== "active" ||
|
||||
!["active", "verifying"].includes(row.endpointStatus)
|
||||
)
|
||||
throw conflict("GitHub bot connection is not active");
|
||||
if (!installationId) throw conflict("Verify the GitHub App installation first");
|
||||
const refs = [appId, privateKey, installationId];
|
||||
const secrets = await db
|
||||
.select({
|
||||
id: companySecrets.id,
|
||||
status: companySecrets.status,
|
||||
latestVersion: companySecrets.latestVersion,
|
||||
updatedAt: companySecrets.updatedAt,
|
||||
bindingId: companySecretBindings.id,
|
||||
bindingPath: companySecretBindings.configPath,
|
||||
bindingSelector: companySecretBindings.versionSelector,
|
||||
bindingUpdatedAt: companySecretBindings.updatedAt,
|
||||
})
|
||||
.from(companySecrets)
|
||||
.leftJoin(
|
||||
companySecretBindings,
|
||||
and(
|
||||
eq(companySecretBindings.secretId, companySecrets.id),
|
||||
eq(companySecretBindings.companyId, companyId),
|
||||
eq(companySecretBindings.targetType, "tool_connection"),
|
||||
eq(companySecretBindings.targetId, row.connectionId),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(companySecrets.companyId, companyId),
|
||||
inArray(companySecrets.id, [...new Set(refs.map((ref) => ref.secretId))]),
|
||||
),
|
||||
);
|
||||
const revision = JSON.stringify([
|
||||
row.connectionId,
|
||||
row.runtimeGeneration ?? 0,
|
||||
refs.map((ref) => {
|
||||
const selector = ref.versionSelector ?? "latest";
|
||||
const secret = secrets.find((candidate) => candidate.id === ref.secretId);
|
||||
const binding = secrets.find(
|
||||
(candidate) =>
|
||||
candidate.id === ref.secretId && candidate.bindingPath === ref.configPath,
|
||||
);
|
||||
return [
|
||||
ref.configPath,
|
||||
ref.secretId,
|
||||
selector,
|
||||
secret
|
||||
? [
|
||||
secret.status,
|
||||
selector === "latest" ? secret.latestVersion : selector,
|
||||
secret.updatedAt.getTime(),
|
||||
]
|
||||
: null,
|
||||
binding
|
||||
? [binding.bindingId, binding.bindingSelector, binding.bindingUpdatedAt?.getTime()]
|
||||
: null,
|
||||
];
|
||||
}),
|
||||
]);
|
||||
return { connectionId: row.connectionId, refs, revision };
|
||||
}
|
||||
|
||||
async function githubBotRepositoryTokenFor(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
endpointId: string,
|
||||
repositoryId: string,
|
||||
fetchImpl: typeof fetch,
|
||||
options: { refresh?: boolean } = {},
|
||||
): Promise<RepositoryToken> {
|
||||
let authority: Awaited<ReturnType<typeof githubBotRepositoryAuthority>>;
|
||||
try {
|
||||
authority = await githubBotRepositoryAuthority(db, companyId, endpointId);
|
||||
} catch (error) {
|
||||
// A connection that cannot mint must not keep its earlier tokens either.
|
||||
forgetEndpointRepositoryTokens(endpointId);
|
||||
throw error;
|
||||
}
|
||||
if (!/^[1-9][0-9]*$/.test(repositoryId) || !Number.isSafeInteger(Number(repositoryId)))
|
||||
throw conflict("Verify the GitHub App installation first");
|
||||
const key = JSON.stringify([
|
||||
companyId,
|
||||
endpointId,
|
||||
repositoryId,
|
||||
GITHUB_REPOSITORY_TOKEN_PERMISSIONS,
|
||||
authority.revision,
|
||||
]);
|
||||
const cached = repositoryTokens.get(key);
|
||||
if (cached && !options.refresh && cached.refreshAt > Date.now()) {
|
||||
rememberRepositoryToken(key, cached);
|
||||
return { token: cached.token, key, cached: true };
|
||||
}
|
||||
if (cached) repositoryTokens.delete(key);
|
||||
// Resolve (and audit) the App's secrets only when a token is actually minted.
|
||||
const secrets = secretService(db);
|
||||
const values: string[] = [];
|
||||
for (const ref of authority.refs)
|
||||
values.push(
|
||||
await secrets.resolveSecretValue(
|
||||
companyId,
|
||||
ref.secretId,
|
||||
ref.versionSelector ?? "latest",
|
||||
{
|
||||
consumerType: "tool_connection",
|
||||
consumerId: authority.connectionId,
|
||||
configPath: ref.configPath,
|
||||
actorType: "system",
|
||||
actorId: null,
|
||||
},
|
||||
),
|
||||
);
|
||||
const [appId, privateKey, installationId] = values;
|
||||
if (!appId || !privateKey)
|
||||
throw conflict("Connect the GitHub App before continuing");
|
||||
if (!installationId) throw conflict("Verify the GitHub App installation first");
|
||||
const issued = await githubBotRequest<{ token?: string; expires_at?: string }>(
|
||||
fetchImpl,
|
||||
githubAppJwt(appId, privateKey),
|
||||
`/app/installations/${encodeURIComponent(installationId)}/access_tokens`,
|
||||
{
|
||||
method: "POST",
|
||||
body: {
|
||||
repository_ids: [Number(repositoryId)],
|
||||
permissions: GITHUB_REPOSITORY_TOKEN_PERMISSIONS,
|
||||
},
|
||||
},
|
||||
);
|
||||
if (!issued.token)
|
||||
throw unprocessable("GitHub did not issue an installation token");
|
||||
const now = Date.now();
|
||||
const refreshAt =
|
||||
Math.min(
|
||||
Date.parse(issued.expires_at ?? ""),
|
||||
now + GITHUB_REPOSITORY_TOKEN_MAX_LIFETIME_MS,
|
||||
) - GITHUB_REPOSITORY_TOKEN_REFRESH_MARGIN_MS;
|
||||
// Without a usable expiry the token serves this call only.
|
||||
if (Number.isFinite(refreshAt) && refreshAt > now)
|
||||
rememberRepositoryToken(key, { token: issued.token, endpointId, refreshAt });
|
||||
return { token: issued.token, key, cached: false };
|
||||
}
|
||||
|
||||
/**
|
||||
* Least-privilege short-lived token restricted to the single task repository.
|
||||
* Reused until shortly before GitHub's expiry while the connection's current
|
||||
* authority is unchanged.
|
||||
*/
|
||||
export async function githubBotRepositoryToken(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
@@ -208,38 +445,60 @@ export async function githubBotRepositoryToken(
|
||||
repositoryId: string,
|
||||
fetchImpl = fetch,
|
||||
) {
|
||||
const result = await githubBotCredentials(db, companyId, endpointId);
|
||||
if (
|
||||
!result.connection.enabled ||
|
||||
result.connection.status !== "active" ||
|
||||
!["active", "verifying"].includes(result.endpoint.status)
|
||||
)
|
||||
throw conflict("GitHub bot connection is not active");
|
||||
if (
|
||||
!result.credentials.installationId ||
|
||||
!/^[1-9][0-9]*$/.test(repositoryId) ||
|
||||
!Number.isSafeInteger(Number(repositoryId))
|
||||
)
|
||||
throw conflict("Verify the GitHub App installation first");
|
||||
const issued = await githubBotRequest<{ token?: string }>(
|
||||
fetchImpl,
|
||||
result.appJwt,
|
||||
`/app/installations/${encodeURIComponent(result.credentials.installationId)}/access_tokens`,
|
||||
{
|
||||
method: "POST",
|
||||
body: {
|
||||
repository_ids: [Number(repositoryId)],
|
||||
permissions: {
|
||||
contents: "read",
|
||||
metadata: "read",
|
||||
issues: "write",
|
||||
pull_requests: "write",
|
||||
checks: "write",
|
||||
},
|
||||
},
|
||||
},
|
||||
return (
|
||||
await githubBotRepositoryTokenFor(db, companyId, endpointId, repositoryId, fetchImpl)
|
||||
).token;
|
||||
}
|
||||
|
||||
export type GitHubBotRepositoryRequest = <T>(
|
||||
path: string,
|
||||
options?: Parameters<typeof githubBotRequest>[3],
|
||||
) => Promise<T>;
|
||||
|
||||
function githubUnauthorized(error: unknown) {
|
||||
return (
|
||||
error instanceof HttpError &&
|
||||
!!error.details &&
|
||||
typeof error.details === "object" &&
|
||||
(error.details as { providerStatus?: unknown }).providerStatus === 401
|
||||
);
|
||||
if (!issued.token)
|
||||
throw unprocessable("GitHub did not issue an installation token");
|
||||
return issued.token;
|
||||
}
|
||||
|
||||
/**
|
||||
* GitHub requests with a repository token. When GitHub rejects a cached token
|
||||
* (401), that token is dropped and the request is sent once more with a newly
|
||||
* minted one. GitHub did not act on the rejected request, so the retry is safe.
|
||||
*/
|
||||
export async function githubBotRepositoryClient(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
endpointId: string,
|
||||
repositoryId: string,
|
||||
fetchImpl = fetch,
|
||||
): Promise<GitHubBotRepositoryRequest> {
|
||||
let current = Promise.resolve(
|
||||
await githubBotRepositoryTokenFor(db, companyId, endpointId, repositoryId, fetchImpl),
|
||||
);
|
||||
return async <T>(path: string, options?: Parameters<typeof githubBotRequest>[3]) => {
|
||||
const used = current;
|
||||
const issued = await used;
|
||||
try {
|
||||
return await githubBotRequest<T>(fetchImpl, issued.token, path, options);
|
||||
} catch (error) {
|
||||
if (!issued.cached || !githubUnauthorized(error)) throw error;
|
||||
if (current === used) {
|
||||
if (repositoryTokens.get(issued.key)?.token === issued.token)
|
||||
repositoryTokens.delete(issued.key);
|
||||
current = githubBotRepositoryTokenFor(
|
||||
db,
|
||||
companyId,
|
||||
endpointId,
|
||||
repositoryId,
|
||||
fetchImpl,
|
||||
{ refresh: true },
|
||||
);
|
||||
}
|
||||
return githubBotRequest<T>(fetchImpl, (await current).token, path, options);
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -13,7 +13,7 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.
|
||||
import { conflict, forbidden, HttpError } from "../errors.js";
|
||||
import { githubChatPrincipalAccess } from "./chat-github-access.js";
|
||||
import { githubAutomaticAdmission } from "./chat-github-events.js";
|
||||
import { githubBotRepositoryToken, githubBotRequest } from "./chat-github-client.js";
|
||||
import { githubBotRepositoryClient } from "./chat-github-client.js";
|
||||
import { withGitHubPublicationLease } from "./chat-github-publication-lease.js";
|
||||
import { projectSafeChatPublicationText } from "./chat-publication-projection.js";
|
||||
import type { GitHubAutomaticPolicyContext } from "./chat-github-review-policy.js";
|
||||
@@ -276,9 +276,9 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
|
||||
["pending", "retry"].includes(fast.publicationState ?? ""))))) return;
|
||||
}
|
||||
}
|
||||
const token = await githubBotRepositoryToken(db, source.endpoint.companyId, source.endpoint.id, source.repositoryId, lease.fetch);
|
||||
const github = await githubBotRepositoryClient(db, source.endpoint.companyId, source.endpoint.id, source.repositoryId, lease.fetch);
|
||||
const prefix = `/repos/${source.repository.split("/").map(encodeURIComponent).join("/")}`;
|
||||
const api: Api = { request: (path, options) => githubBotRequest(lease.fetch, token, `${prefix}${path}`, options) };
|
||||
const api: Api = { request: (path, options) => github(`${prefix}${path}`, options) };
|
||||
if (onlyExisting && !(await findResponseComment(source, api))) {
|
||||
const existing = await stageResponse(db, source);
|
||||
await lease.commit(async tx => { await tx.update(chatActions).set({ status: "cancelled", result: { final: true, code: "run_already_answered" }, updatedAt: new Date() }).where(eq(chatActions.id, existing.id)); });
|
||||
|
||||
@@ -43,7 +43,7 @@ import {
|
||||
} from "@paperclipai/shared";
|
||||
import { HttpError, conflict, forbidden, notFound } from "../errors.js";
|
||||
import {
|
||||
githubBotRepositoryToken,
|
||||
githubBotRepositoryClient,
|
||||
githubBotRequest,
|
||||
} from "./chat-github-client.js";
|
||||
import { githubChatPrincipalAccess } from "./chat-github-access.js";
|
||||
@@ -316,7 +316,7 @@ export function githubChatReviewService(db: Db, fetchImpl = fetch) {
|
||||
source: Awaited<ReturnType<typeof scope>>,
|
||||
requestFetch = fetchImpl,
|
||||
) {
|
||||
const token = await githubBotRepositoryToken(
|
||||
const github = await githubBotRepositoryClient(
|
||||
db,
|
||||
source.endpoint.companyId,
|
||||
source.endpoint.id,
|
||||
@@ -330,7 +330,7 @@ export function githubChatReviewService(db: Db, fetchImpl = fetch) {
|
||||
path: string,
|
||||
options?: Parameters<typeof githubBotRequest>[3],
|
||||
) =>
|
||||
githubBotRequest<T>(requestFetch, token, `${prefix}${path}`, options),
|
||||
github<T>(`${prefix}${path}`, options),
|
||||
};
|
||||
}
|
||||
async function reviewForHead(
|
||||
|
||||
Reference in new issue
Block a user