mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 04:02:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connections need durable credentials that survive authorization and token rotation. > - Storage, registration, refresh coordination and credential selection lived in the general tool service. > - Each has a different ownership boundary: secrets, process-wide registration, database leases and per-service refresh work. > - This pull request gives those responsibilities named modules and passes their narrow capabilities to consumers. > - The benefit is visible credential ownership while retaining current concurrency and failure behavior. ## Linked Issues or Issue Description Depends on #17159. Earlier stack: #17144 → #17146 → #17147 → #17155 → #17157 → #17159 → this PR. **What existing behavior does this improve?** Maintenance of OAuth credential persistence, refresh and provider credential access. **Current behavior** The 14,877-line service owns registration flights, refresh leases, secret writes and provider selection alongside app setup. **Proposed behavior** Dedicated owners handle credential storage, client registration, refresh leases, connection/grant refresh, GitHub continuity and Vercel credential access. Credential selection composes those owners. Authorization callbacks remain in the coordinator for the next extraction. **Reason and benefit** The entry point falls to 12,019 lines, a reduction of 2,858. Lease acquisition and cleanup can be reviewed together, independently of callback completion. ## What Changed - Extract credential owners and shared single-flight/connection-link primitives. The largest new module is 650 lines. - Preserve process-wide client-registration flights, per-service refresh maps and GitHub sweep state. - Preserve live cloud-connector lookup, options and environment reads; use the existing secret client and transaction-scoped clients. - Extend the real local HTTP/PostgreSQL OAuth fixture with concurrent refresh across two service instances and authenticated use of rotated credentials. ## Verification - The two extended HTTP integration cases pass before extraction for public and client-secret-basic clients. - After extraction, all 470 public service and HTTP integration cases pass, including refresh failure, abandoned leases, secret redaction and membership revocation cases. The fixture was then strengthened after review: a second expiry must use the persisted rotated refresh token, and the provider rejects the old token. Both cases pass before and after extraction; server typecheck also passes. - Server typecheck, module-boundary gate and diff whitespace checks pass. - All 233 pre-existing function/method bodies match by normalized syntax comparison. - Full typecheck/build will run at the stack handoff. The earlier interrupted local full suite remains unverified; all current-head CI gates pass, with Greptile 5/5 and no open review findings. - Prior-head CI failure retained: run 39708550066, general server shard 2/12, `native-session-executor.test.ts` case “preserves a real listening process and its marker when a health RPC times out” raised `runner_remote_process_identity_unavailable` / `process_identity_changed`. Both the fixture and runner implementation are unchanged. The isolated case passes without changing assertions or timeouts. The fresh full CI run passes for the test-only review fix; the prior failure remains recorded here. ## Risks - Changing state lifetime could permit duplicate refreshes or registrations. Existing concurrency cases and the real HTTP fixture exercise these paths. - A transaction-scoped secret write must stay inside its authority lock. Function bodies, await ordering and client arguments are preserved. - The fixture proves assembled local HTTP/database behavior, not live third-party qualification. - No schema, public API, policy or scheduling change is intended. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
304 lines
11 KiB
TypeScript
304 lines
11 KiB
TypeScript
import type { ActorInfo } from "./types.js";
|
|
import { loadGitHubGrantMetadata } from "./github-access-discovery.js";
|
|
import { measureRunPhase } from "../execution-preparation-timing.js";
|
|
import { measureRunTransaction } from "../execution-transaction-timing.js";
|
|
import { and, eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { connectionGrants, toolConnections } from "@paperclipai/db";
|
|
import { conflict, HttpError, notFound, unprocessable } from "../../errors.js";
|
|
import { type PaperclipCloudConnector } from "../paperclip-cloud-connector.js";
|
|
|
|
import type { createOAuthGrantRefresh } from "./oauth/grant-refresh.js";
|
|
import type { OAuthCredentialStore } from "./oauth/credential-store.js";
|
|
function asRecord(value: unknown): Record<string, unknown> {
|
|
if (value && typeof value === "object" && !Array.isArray(value))
|
|
return value as Record<string, unknown>;
|
|
return {};
|
|
}
|
|
|
|
type GitHubContinuityDependencies = {
|
|
now: () => Date;
|
|
currentCloudConnector: () => PaperclipCloudConnector | null;
|
|
refreshOAuthGrantCredentials: ReturnType<typeof createOAuthGrantRefresh>;
|
|
resolveOAuthGrantSecret: OAuthCredentialStore["resolveOAuthGrantSecret"];
|
|
};
|
|
|
|
export function createGitHubGrantContinuity(db: Db, { now, currentCloudConnector, refreshOAuthGrantCredentials, resolveOAuthGrantSecret }: GitHubContinuityDependencies) {
|
|
let nextGitHubContinuitySweepAt = 0;
|
|
async function refreshManagedGitHubGrantAccess(
|
|
connection: typeof toolConnections.$inferSelect,
|
|
initialGrant: typeof connectionGrants.$inferSelect,
|
|
actor?: ActorInfo,
|
|
) {
|
|
return measureRunPhase("connectors.github.refresh", async () => {
|
|
try {
|
|
return await refreshManagedGitHubGrantAccessOnce(
|
|
connection,
|
|
initialGrant,
|
|
actor,
|
|
);
|
|
} catch (error) {
|
|
if (
|
|
!(error instanceof HttpError) ||
|
|
asRecord(error.details).code !== "github_access_changed"
|
|
)
|
|
throw error;
|
|
const [current] = await measureRunPhase("connectors.github.grants", () => db
|
|
.select()
|
|
.from(connectionGrants)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.id, initialGrant.id),
|
|
eq(connectionGrants.companyId, connection.companyId),
|
|
eq(connectionGrants.connectionId, connection.id),
|
|
),
|
|
));
|
|
if (!current || current.status !== "active") throw error;
|
|
return refreshManagedGitHubGrantAccessOnce(connection, current, actor);
|
|
}
|
|
});
|
|
}
|
|
|
|
async function refreshManagedGitHubGrantAccessOnce(
|
|
connection: typeof toolConnections.$inferSelect,
|
|
initialGrant: typeof connectionGrants.$inferSelect,
|
|
actor?: ActorInfo,
|
|
) {
|
|
let grant = await measureRunPhase("connectors.github.credentials", () => refreshOAuthGrantCredentials({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
grantId: initialGrant.id,
|
|
actor,
|
|
}));
|
|
const resolveAccessToken = async () => measureRunPhase("connectors.github.credentials", async () => {
|
|
const accessRef = grant.credentialSecretRefs.find(
|
|
(ref) => ref.configPath === "oauth.access_token",
|
|
);
|
|
if (!accessRef)
|
|
throw unprocessable("GitHub authorization must be reconnected", {
|
|
code: "oauth_reauthorization_required",
|
|
});
|
|
return (
|
|
await resolveOAuthGrantSecret(
|
|
connection,
|
|
grant,
|
|
accessRef,
|
|
actor,
|
|
undefined,
|
|
)
|
|
).value;
|
|
});
|
|
let metadata;
|
|
let accessToken = await resolveAccessToken();
|
|
try {
|
|
metadata = await measureRunPhase("connectors.github.repositories", () => loadGitHubGrantMetadata(
|
|
accessToken,
|
|
fetch,
|
|
grant.providerTenant?.github?.appSlug,
|
|
));
|
|
} catch (error) {
|
|
const providerCode =
|
|
error instanceof HttpError ? asRecord(error.details).code : null;
|
|
if (providerCode !== "oauth_reauthorization_required") throw error;
|
|
// GitHub may invalidate an access token before its recorded expiry. If an
|
|
// expiring token pair exists, rotate it once under the same durable lease
|
|
// and CAS path, then repeat /user before surfacing reconnect-required.
|
|
grant = await measureRunPhase("connectors.github.credentials", () => refreshOAuthGrantCredentials({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
grantId: initialGrant.id,
|
|
forceRefresh: true,
|
|
actor,
|
|
}));
|
|
accessToken = await resolveAccessToken();
|
|
try {
|
|
metadata = await measureRunPhase("connectors.github.repositories", () => loadGitHubGrantMetadata(
|
|
accessToken,
|
|
fetch,
|
|
grant.providerTenant?.github?.appSlug,
|
|
));
|
|
} catch (retryError) {
|
|
const retryCode =
|
|
retryError instanceof HttpError
|
|
? asRecord(retryError.details).code
|
|
: null;
|
|
if (retryCode === "oauth_reauthorization_required") {
|
|
await measureRunPhase("connectors.github.grant_write", () => db
|
|
.update(connectionGrants)
|
|
.set({ status: "needs_reauthorization", updatedAt: now() })
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.id, grant.id),
|
|
eq(connectionGrants.companyId, grant.companyId),
|
|
),
|
|
));
|
|
}
|
|
throw retryError;
|
|
}
|
|
}
|
|
const { updated, previousGitHub } = await measureRunTransaction(db, "connectors.github.refresh", async (tx) => {
|
|
const [currentGrant] = await measureRunPhase("connectors.github.grant_lock", () => tx
|
|
.select()
|
|
.from(connectionGrants)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.id, grant.id),
|
|
eq(connectionGrants.companyId, grant.companyId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1));
|
|
if (!currentGrant || currentGrant.status === "revoked")
|
|
throw notFound("GitHub authorization not found");
|
|
const previousGitHub = currentGrant.providerTenant?.github;
|
|
const initialGitHub = grant.providerTenant?.github;
|
|
// No lock is held during provider requests. Reject a snapshot if another
|
|
// refresh or webhook changed access while those requests were in flight.
|
|
if (
|
|
previousGitHub?.accessRevision !== initialGitHub?.accessRevision ||
|
|
previousGitHub?.lastWebhookAt !== initialGitHub?.lastWebhookAt ||
|
|
previousGitHub?.lastAccessRefreshAt !==
|
|
initialGitHub?.lastAccessRefreshAt
|
|
) {
|
|
throw conflict("GitHub access changed during refresh. Try again.", {
|
|
code: "github_access_changed",
|
|
});
|
|
}
|
|
const providerTenant = {
|
|
...(currentGrant.providerTenant ?? {}),
|
|
github: {
|
|
...metadata,
|
|
...(previousGitHub?.lastWebhookAt
|
|
? { lastWebhookAt: previousGitHub.lastWebhookAt }
|
|
: {}),
|
|
webhookHealth:
|
|
previousGitHub?.webhookHealth ?? metadata.webhookHealth,
|
|
},
|
|
};
|
|
const [updated] = await measureRunPhase("connectors.github.grant_write", () => tx
|
|
.update(connectionGrants)
|
|
.set({
|
|
providerTenant,
|
|
status: "active",
|
|
updatedAt: now(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.id, grant.id),
|
|
eq(connectionGrants.companyId, grant.companyId),
|
|
),
|
|
)
|
|
.returning());
|
|
if (!updated) throw notFound("GitHub authorization not found");
|
|
return { updated, previousGitHub };
|
|
});
|
|
|
|
const cloudConnector = currentCloudConnector();
|
|
const subject =
|
|
updated.kind === "agent" && updated.subjectAgentId
|
|
? `agent:${updated.subjectAgentId}`
|
|
: updated.kind === "user" && updated.subjectUserId
|
|
? updated.subjectUserId
|
|
: null;
|
|
if (cloudConnector && subject) {
|
|
const previous = new Set(previousGitHub?.installationIds ?? []);
|
|
const current = new Set(metadata.installationIds);
|
|
await measureRunPhase("connectors.github.webhooks", () => Promise.all([
|
|
...metadata.installationIds.map((installationId) =>
|
|
cloudConnector.setWebhookBinding({
|
|
subject,
|
|
companyId: connection.companyId,
|
|
id: `${updated.id}_${installationId}`,
|
|
installationId,
|
|
connectionId: connection.id,
|
|
grantId: updated.id,
|
|
active: true,
|
|
accessToken,
|
|
}),
|
|
),
|
|
...[...previous]
|
|
.filter((installationId) => !current.has(installationId))
|
|
.map((installationId) =>
|
|
cloudConnector.setWebhookBinding({
|
|
subject,
|
|
companyId: connection.companyId,
|
|
id: `${updated.id}_${installationId}`,
|
|
installationId,
|
|
connectionId: connection.id,
|
|
grantId: updated.id,
|
|
active: false,
|
|
}),
|
|
),
|
|
]));
|
|
}
|
|
return updated;
|
|
}
|
|
|
|
async function sweepGitHubConnectionContinuity() {
|
|
if (now().getTime() < nextGitHubContinuitySweepAt) {
|
|
return { checked: 0, due: 0, refreshed: 0, failed: 0 };
|
|
}
|
|
nextGitHubContinuitySweepAt = now().getTime() + 60 * 60_000;
|
|
const cutoff = now().getTime() - 30 * 24 * 60 * 60_000;
|
|
const rows = await db
|
|
.select({ grant: connectionGrants, connection: toolConnections })
|
|
.from(connectionGrants)
|
|
.innerJoin(
|
|
toolConnections,
|
|
and(
|
|
eq(toolConnections.id, connectionGrants.connectionId),
|
|
eq(toolConnections.companyId, connectionGrants.companyId),
|
|
),
|
|
)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.status, "active"),
|
|
eq(toolConnections.status, "active"),
|
|
eq(toolConnections.enabled, true),
|
|
),
|
|
);
|
|
const due = rows.filter(({ grant, connection }) => {
|
|
const config = asRecord(connection.config);
|
|
const oauth = asRecord(config.oauth);
|
|
if (
|
|
config.sourceTemplateKey !== "github" ||
|
|
oauth.connectorProfile !== "github.code" ||
|
|
!grant.providerTenant?.github
|
|
)
|
|
return false;
|
|
const expiresAt = grant.providerTenant.oauth?.accessTokenExpiresAt;
|
|
const refreshedAt = grant.providerTenant.oauth?.refreshedAt
|
|
? Date.parse(grant.providerTenant.oauth.refreshedAt)
|
|
: Number.NaN;
|
|
const accessCheckedAt = grant.providerTenant.github.lastAccessRefreshAt
|
|
? Date.parse(grant.providerTenant.github.lastAccessRefreshAt)
|
|
: Number.NaN;
|
|
if (typeof expiresAt === "string") {
|
|
const expiry = Date.parse(expiresAt);
|
|
return (
|
|
!Number.isFinite(expiry) ||
|
|
expiry <= now().getTime() + 60 * 60_000 ||
|
|
!Number.isFinite(refreshedAt) ||
|
|
refreshedAt <= cutoff
|
|
);
|
|
}
|
|
return !Number.isFinite(accessCheckedAt) || accessCheckedAt <= cutoff;
|
|
});
|
|
let refreshed = 0;
|
|
let failed = 0;
|
|
for (const row of due) {
|
|
try {
|
|
await refreshManagedGitHubGrantAccess(row.connection, row.grant, {
|
|
actorType: "system",
|
|
actorId: "github-connection-continuity",
|
|
});
|
|
refreshed += 1;
|
|
} catch {
|
|
failed += 1;
|
|
}
|
|
}
|
|
return { checked: rows.length, due: due.length, refreshed, failed };
|
|
}
|
|
return { refreshManagedGitHubGrantAccess, sweepGitHubConnectionContinuity };
|
|
}
|