mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connections need durable credentials that survive authorization and token rotation. > - Storage, registration, refresh coordination and credential selection lived in the general tool service. > - Each has a different ownership boundary: secrets, process-wide registration, database leases and per-service refresh work. > - This pull request gives those responsibilities named modules and passes their narrow capabilities to consumers. > - The benefit is visible credential ownership while retaining current concurrency and failure behavior. ## Linked Issues or Issue Description Depends on #17159. Earlier stack: #17144 → #17146 → #17147 → #17155 → #17157 → #17159 → this PR. **What existing behavior does this improve?** Maintenance of OAuth credential persistence, refresh and provider credential access. **Current behavior** The 14,877-line service owns registration flights, refresh leases, secret writes and provider selection alongside app setup. **Proposed behavior** Dedicated owners handle credential storage, client registration, refresh leases, connection/grant refresh, GitHub continuity and Vercel credential access. Credential selection composes those owners. Authorization callbacks remain in the coordinator for the next extraction. **Reason and benefit** The entry point falls to 12,019 lines, a reduction of 2,858. Lease acquisition and cleanup can be reviewed together, independently of callback completion. ## What Changed - Extract credential owners and shared single-flight/connection-link primitives. The largest new module is 650 lines. - Preserve process-wide client-registration flights, per-service refresh maps and GitHub sweep state. - Preserve live cloud-connector lookup, options and environment reads; use the existing secret client and transaction-scoped clients. - Extend the real local HTTP/PostgreSQL OAuth fixture with concurrent refresh across two service instances and authenticated use of rotated credentials. ## Verification - The two extended HTTP integration cases pass before extraction for public and client-secret-basic clients. - After extraction, all 470 public service and HTTP integration cases pass, including refresh failure, abandoned leases, secret redaction and membership revocation cases. The fixture was then strengthened after review: a second expiry must use the persisted rotated refresh token, and the provider rejects the old token. Both cases pass before and after extraction; server typecheck also passes. - Server typecheck, module-boundary gate and diff whitespace checks pass. - All 233 pre-existing function/method bodies match by normalized syntax comparison. - Full typecheck/build will run at the stack handoff. The earlier interrupted local full suite remains unverified; all current-head CI gates pass, with Greptile 5/5 and no open review findings. - Prior-head CI failure retained: run 39708550066, general server shard 2/12, `native-session-executor.test.ts` case “preserves a real listening process and its marker when a health RPC times out” raised `runner_remote_process_identity_unavailable` / `process_identity_changed`. Both the fixture and runner implementation are unchanged. The isolated case passes without changing assertions or timeouts. The fresh full CI run passes for the test-only review fix; the prior failure remains recorded here. ## Risks - Changing state lifetime could permit duplicate refreshes or registrations. Existing concurrency cases and the real HTTP fixture exercise these paths. - A transaction-scoped secret write must stay inside its authority lock. Function bodies, await ordering and client arguments are preserved. - The fixture proves assembled local HTTP/database behavior, not live third-party qualification. - No schema, public API, policy or scheduling change is intended. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
302 lines
12 KiB
TypeScript
302 lines
12 KiB
TypeScript
import type { ActorInfo } from "./types.js";
|
|
import { oauthConfig } from "./oauth/metadata.js";
|
|
import { remoteEndpoint } from "./connection-methods.js";
|
|
import { and, eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { connectionGrants, toolConnections } from "@paperclipai/db";
|
|
import { badRequest, HttpError, unprocessable } from "../../errors.js";
|
|
import { parseRemoteHttpEndpoint } from "../remote-http-endpoint-guard.js";
|
|
import { REMOTE_URL_SECRET_CONFIG_PATH, remoteUrlCredentialMatchesPublicUrl } from "../remote-url-credentials.js";
|
|
import { secretService } from "../secrets.js";
|
|
import { connectionCredentialConfigPath as credentialRefConfigPath, connectionGrantCredentialRef } from "../connection-credentials.js";
|
|
|
|
import type { ToolAccessAudit } from "./audit.js";
|
|
import type { VercelCredentialService } from "./vercel-credentials.js";
|
|
import type { OAuthCredentialStore } from "./oauth/credential-store.js";
|
|
import type { createLegacyOAuthRefresh } from "./oauth/connection-refresh.js";
|
|
import type { createOAuthGrantRefresh } from "./oauth/grant-refresh.js";
|
|
import { credentialScope } from "./oauth/client-settings.js";
|
|
import { connectionSetupUrl, connectionReconnectUrl } from "./identifiers.js";
|
|
function asRecord(value: unknown): Record<string, unknown> {
|
|
if (value && typeof value === "object" && !Array.isArray(value))
|
|
return value as Record<string, unknown>;
|
|
return {};
|
|
}
|
|
|
|
type CredentialAccessDependencies = {
|
|
secrets: ReturnType<typeof secretService>;
|
|
audit: ToolAccessAudit;
|
|
vercel: VercelCredentialService;
|
|
resolveOAuthGrantSecret: OAuthCredentialStore["resolveOAuthGrantSecret"];
|
|
maybeRefreshOAuthCredentials: ReturnType<typeof createLegacyOAuthRefresh>;
|
|
refreshOAuthGrantCredentials: ReturnType<typeof createOAuthGrantRefresh>;
|
|
};
|
|
|
|
export function createConnectionCredentialAccess(db: Db, { secrets, audit, vercel, resolveOAuthGrantSecret, maybeRefreshOAuthCredentials, refreshOAuthGrantCredentials }: CredentialAccessDependencies) {
|
|
const { resolveVercelCredentialHeaders, vercelGrantForConnection } = vercel;
|
|
async function vaultGrantForConnection(
|
|
connection: typeof toolConnections.$inferSelect,
|
|
actor?: ActorInfo,
|
|
): Promise<typeof connectionGrants.$inferSelect | null> {
|
|
const actorUserId =
|
|
actor?.actorType === "user" ? (actor.actorId ?? null) : null;
|
|
if (actorUserId) {
|
|
const [personal] = await db
|
|
.select()
|
|
.from(connectionGrants)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.companyId, connection.companyId),
|
|
eq(connectionGrants.connectionId, connection.id),
|
|
eq(connectionGrants.kind, "user"),
|
|
eq(connectionGrants.subjectUserId, actorUserId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (personal) {
|
|
if (personal.status !== "active") {
|
|
throw unprocessable("OAuth authorization must be reconnected", {
|
|
code: "oauth_reauthorization_required",
|
|
setupUrl: connectionSetupUrl(connection),
|
|
reconnectUrl: connectionReconnectUrl(connection),
|
|
});
|
|
}
|
|
return personal;
|
|
}
|
|
}
|
|
if (!actorUserId && connection.credentialPolicy === "per_user") {
|
|
// A personal-only connection is intentionally fixed to one identity.
|
|
// Background health/catalog checks have no acting user, but may safely
|
|
// exercise that sole owner-bound grant without turning it into a shared
|
|
// credential or making it available to a different caller.
|
|
const personalGrants = await db
|
|
.select()
|
|
.from(connectionGrants)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.companyId, connection.companyId),
|
|
eq(connectionGrants.connectionId, connection.id),
|
|
eq(connectionGrants.kind, "user"),
|
|
eq(connectionGrants.status, "active"),
|
|
),
|
|
)
|
|
.limit(2);
|
|
if (personalGrants.length === 1) return personalGrants[0]!;
|
|
}
|
|
if (connection.credentialPolicy === "per_user") {
|
|
throw unprocessable(
|
|
"This connection needs the current user's authorization",
|
|
{
|
|
code: "user_authorization_required",
|
|
setupUrl: connectionSetupUrl(connection),
|
|
reconnectUrl: connectionReconnectUrl(connection),
|
|
},
|
|
);
|
|
}
|
|
const [organization] = await db
|
|
.select()
|
|
.from(connectionGrants)
|
|
.where(
|
|
and(
|
|
eq(connectionGrants.companyId, connection.companyId),
|
|
eq(connectionGrants.connectionId, connection.id),
|
|
eq(connectionGrants.kind, "organization"),
|
|
eq(connectionGrants.isDefault, true),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (organization?.status === "active") return organization;
|
|
return null;
|
|
}
|
|
|
|
async function resolveCredentialHeaders(
|
|
connection: typeof toolConnections.$inferSelect,
|
|
actor?: ActorInfo,
|
|
options: { forceRefresh?: boolean } = {},
|
|
): Promise<Record<string, string>> {
|
|
if (connection.credentialSource === "vercel_connect") {
|
|
const grant = await vercelGrantForConnection(connection, actor);
|
|
return resolveVercelCredentialHeaders(connection, grant, options);
|
|
}
|
|
let grant: typeof connectionGrants.$inferSelect | null = null;
|
|
try {
|
|
if (
|
|
connection.authKind === "oauth" ||
|
|
connection.credentialPolicy !== "shared"
|
|
) {
|
|
grant = await vaultGrantForConnection(connection, actor);
|
|
}
|
|
if (connection.authKind === "oauth" && grant) {
|
|
grant = await refreshOAuthGrantCredentials({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
grantId: grant.id,
|
|
forceRefresh: options.forceRefresh,
|
|
actor,
|
|
});
|
|
} else {
|
|
connection = await maybeRefreshOAuthCredentials(connection, actor);
|
|
}
|
|
} catch (error) {
|
|
const scope = credentialScope(connection);
|
|
await audit({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
action: "tool_connection.credential_resolution",
|
|
outcome: "failure",
|
|
reasonCode:
|
|
error instanceof HttpError
|
|
? String(asRecord(error.details).code ?? "oauth_refresh_failed")
|
|
: "oauth_refresh_failed",
|
|
details: {
|
|
credentialCount: connection.credentialRefs.length,
|
|
credentialSecretRefCount: connection.credentialSecretRefs.length,
|
|
credentialScopeType: scope.type,
|
|
credentialScopeHash: scope.hash,
|
|
setupUrl: connectionSetupUrl(connection),
|
|
reconnectUrl: connectionReconnectUrl(connection),
|
|
},
|
|
});
|
|
throw error;
|
|
}
|
|
const headers: Record<string, string> = {};
|
|
const scope = credentialScope(connection);
|
|
for (const ref of connection.credentialRefs) {
|
|
if (ref.placement !== "header") continue;
|
|
let value: string;
|
|
const configPath = credentialRefConfigPath(ref);
|
|
try {
|
|
const grantRef = grant ? connectionGrantCredentialRef(grant, ref) : undefined;
|
|
if (grant && !grantRef) throw unprocessable("Your credential is missing. Reconnect this connection.", {
|
|
code: "grant_credential_invalid", connectionId: connection.id, grantId: grant.id, credential: configPath,
|
|
});
|
|
value =
|
|
grantRef && grant
|
|
? (
|
|
await resolveOAuthGrantSecret(
|
|
connection,
|
|
grant,
|
|
grantRef,
|
|
actor,
|
|
undefined,
|
|
)
|
|
).value
|
|
: await secrets.resolveSecretValue(
|
|
connection.companyId,
|
|
ref.secretId,
|
|
ref.version ?? "latest",
|
|
{
|
|
consumerType: "tool_connection",
|
|
consumerId: connection.id,
|
|
configPath,
|
|
actorType: "system",
|
|
},
|
|
);
|
|
} catch (error) {
|
|
await audit({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
action: "tool_connection.credential_resolution",
|
|
outcome: "failure",
|
|
reasonCode:
|
|
error instanceof HttpError
|
|
? String(
|
|
asRecord(error.details).code ?? "secret_resolution_failed",
|
|
)
|
|
: "secret_resolution_failed",
|
|
details: {
|
|
credentialCount: connection.credentialRefs.length,
|
|
credentialScopeType: scope.type,
|
|
credentialScopeHash: scope.hash,
|
|
},
|
|
});
|
|
throw error;
|
|
}
|
|
if (ref.placement === "header") {
|
|
headers[ref.key] = `${ref.prefix ?? ""}${value}`;
|
|
}
|
|
}
|
|
const oauthAccessRef = grant?.credentialSecretRefs.find(
|
|
(ref) => ref.configPath === "oauth.access_token",
|
|
);
|
|
if (oauthAccessRef && headers.Authorization === undefined) {
|
|
headers.Authorization = `Bearer ${
|
|
(
|
|
await resolveOAuthGrantSecret(
|
|
connection,
|
|
grant!,
|
|
oauthAccessRef,
|
|
actor,
|
|
undefined,
|
|
)
|
|
).value
|
|
}`;
|
|
}
|
|
if (
|
|
connection.credentialRefs.length > 0 ||
|
|
connection.credentialSecretRefs.length > 0 ||
|
|
Object.keys(oauthConfig(connection)).length > 0
|
|
) {
|
|
await audit({
|
|
companyId: connection.companyId,
|
|
connectionId: connection.id,
|
|
action: "tool_connection.credential_resolution",
|
|
outcome: "success",
|
|
details: {
|
|
credentialCount: connection.credentialRefs.length,
|
|
credentialSecretRefCount: connection.credentialSecretRefs.length,
|
|
credentialScopeType: scope.type,
|
|
credentialScopeHash: scope.hash,
|
|
},
|
|
});
|
|
}
|
|
return headers;
|
|
}
|
|
|
|
async function resolvedRemoteEndpoint(
|
|
connection: typeof toolConnections.$inferSelect,
|
|
actor?: ActorInfo,
|
|
): Promise<string> {
|
|
const publicEndpoint = remoteEndpoint(connection.config);
|
|
const ref = connection.credentialRefs.find(
|
|
(candidate) => candidate.placement === "url",
|
|
);
|
|
if (!ref) return publicEndpoint;
|
|
let value: string;
|
|
try {
|
|
const grant = await vaultGrantForConnection(connection, actor);
|
|
const grantRef = grant ? connectionGrantCredentialRef(grant, ref) : undefined;
|
|
if (grant && !grantRef) throw unprocessable("Your MCP URL credential is missing. Reconnect this connection.", {
|
|
code: "grant_credential_invalid", connectionId: connection.id, grantId: grant.id,
|
|
});
|
|
value = grant && grantRef
|
|
? (await resolveOAuthGrantSecret(connection, grant, grantRef, actor, undefined)).value
|
|
: await secrets.resolveSecretValue(connection.companyId, ref.secretId, ref.version ?? "latest", {
|
|
consumerType: "tool_connection", consumerId: connection.id,
|
|
configPath: REMOTE_URL_SECRET_CONFIG_PATH,
|
|
actorType: actor?.actorType ?? "system", actorId: actor?.actorId ?? null,
|
|
});
|
|
} catch (error) {
|
|
if (error instanceof HttpError) throw error;
|
|
throw unprocessable("Your MCP URL credential could not be resolved. Reconnect this connection.", {
|
|
code: "mcp_remote_missing_secret", connectionId: connection.id, credential: REMOTE_URL_SECRET_CONFIG_PATH,
|
|
});
|
|
}
|
|
if (!remoteUrlCredentialMatchesPublicUrl(publicEndpoint, value)) {
|
|
throw unprocessable(
|
|
"The stored MCP URL credential no longer matches this connection.",
|
|
{
|
|
code: "mcp_remote_url_credential_mismatch",
|
|
connectionId: connection.id,
|
|
},
|
|
);
|
|
}
|
|
return parseRemoteHttpEndpoint(value, (message, code) =>
|
|
badRequest(message, { code }),
|
|
).toString();
|
|
}
|
|
return { vaultGrantForConnection, resolveCredentialHeaders, resolvedRemoteEndpoint };
|
|
}
|
|
|
|
export type ConnectionCredentialAccess = ReturnType<typeof createConnectionCredentialAccess>;
|