mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
feat(workspaces): sign the workspace login handoff and gate readiness (#11671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed worktree services run isolated Paperclip instances with cloned databases. > - A reachable service was reported as ready even when its database, runtime identity, or login path was not usable. > - The first candidate added verified database seeding and managed repair in #11665. > - This pull request consolidates that candidate with signed login handoff and a complete readiness contract. > - Post-QA fixes close five defects in repair identity, repair responses, UI retry, seed journal handling, and seed-source trust. > - The benefit is a workspace that either opens safely or reports one accurate recovery action. ## Linked Issues or Issue Description No public GitHub issue exists for this work, so the problem is described here. **What happened** Managed workspace URLs could return HTTP 200 and report ready while login failed. QA also found cases where repair used the wrong instance identity, returned a generic error, left the UI stuck, rejected a safe journal lag, or trusted a mutable workspace manifest. **Expected behavior** Opening a ready workspace signs the board user in to the correct isolated instance. Provisioning and repair use a registered source and report a structured recovery state. **Actual behavior** Entry depended on a password copied into the clone. Several failure paths could publish stale readiness, hide the repair precondition, or trust state that the workspace could modify. **Additional context** This pull request includes the commits first published in #11665. That pull request keeps the original base head for review history. This consolidated pull request is the merge candidate. Related open readiness work includes #11575 and #11621. ## What Changed - Adds a short-lived, signed, single-use login ticket. It binds the user, workspace, instance, and runtime origin. - Exchanges the ticket through Better Auth. It creates the session and cookie through the supported adapter path. - Adds protected workspace readiness fields for the database, clone data, login handoff, seed phase, and runtime identity. - Fails readiness closed when the guest has no company or execution-workspace binding. - Binds ticket issuance to the exact cloned user and active company membership selected for the handoff. - Verifies every current active board identity through the exact-user handoff before publication or reuse. - Gates managed runtime publication on the readiness contract and the recorded worktree instance identity. - Refreshes runtime work products from the live runtime row after a port change. - Adds one workspace access card with ready, degraded, repairing, and failed states. - Uses the runtime response identity for repair. It returns structured repair precondition errors. - Lets a valid source journal lag converge during provisioning. - Binds seed and repair manifests to a source registered outside the agent-writable worktree. - Clears recovered UI errors so a successful retry can open the workspace. - Makes runtime tests register canonical sources and avoid ports owned by live host listeners. - Keeps Vitest on source suites when compiled `dist` trees exist. - Isolates CLI and adapter tests from ambient AWS and runtime API environment variables. - Preserves a 404 response for cross-company workspace ID lookups before runtime authorization. - Makes concurrent single-flight coverage independent of path-canonicalization scheduling order. ## Verification The following checks passed on the integrated head: ```sh pnpm -r typecheck pnpm build pnpm check:token-gates pnpm --filter @paperclipai/db check:migrations ``` - The server source lane passed 420 files and 4,953 tests. Five tests were skipped. - The CLI lane passed 57 files and 385 tests. - The database lane passed 26 files and 97 tests. - The shared package passed 58 files and 506 tests. - The adapter utility lane passed 640 tests. Four tests were skipped. - The Claude adapter passed 220 tests. One test was skipped. - The Codex adapter passed 323 tests. - The OpenClaw adapter passed 13 tests. - The OpenCode adapter passed 42 tests. - The plugin SDK passed 45 tests. - The workspace runtime suite passed 124 tests. - The caller-scoped readiness and handoff suite passed 52 tests. - The workspace provisioning shell suite passed 7 tests. - The runtime exposure suite passed 17 tests while live host mappings occupied fixed test ports. - `git diff --check` passed and the worktree is clean. The serialized route lane will run in GitHub CI with its normal shards. No deployment or active-workspace migration was performed. ## Risks - This is a medium-risk authentication and runtime-readiness change. - The login ticket uses exact origin, workspace, instance, and user binding. It has a short expiry and a one-time nonce. - Runtime publication is stricter. A real readiness, identity, per-user handoff, or control-plane database disagreement now blocks publication. - This pull request supersedes #11665 as the merge candidate. Close #11665 after this pull request merges. - No new database migration is included. The lockfile and workflow files are unchanged. - Deployment and active-workspace migration are intentionally outside this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5 (`claude-opus-5[1m]`), 1M context, extended thinking, tool use, and code execution produced the main candidate. OpenAI GPT-5 (`gpt-5`) through Codex, with agentic reasoning, tool use, and code execution, integrated the post-QA fixes and hardened the test gates. The Codex context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
6aaef2998f
commit
a2bf936f9a
59 files changed
+7754
-243
No files matched your search
@@ -12,6 +12,16 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import type { Config } from "../config.js";
|
||||
import { resolvePaperclipInstanceId } from "../home-paths.js";
|
||||
import {
|
||||
workspaceLoginHandoffPlugin,
|
||||
type WorkspaceHandoffExpectedIdentity,
|
||||
} from "./workspace-login-handoff-plugin.js";
|
||||
import {
|
||||
normalizeWorkspaceHandoffOrigin,
|
||||
resolveWorkspaceHandoffLocalCompanyId,
|
||||
resolveWorkspaceHandoffLocalKey,
|
||||
resolveWorkspaceHandoffLocalWorkspaceId,
|
||||
} from "./workspace-login-handoff.js";
|
||||
|
||||
export type BetterAuthSessionUser = {
|
||||
id: string;
|
||||
@@ -144,6 +154,34 @@ export function deriveAuthTrustedOrigins(config: Config, opts?: { listenPort?: n
|
||||
return Array.from(trustedOrigins);
|
||||
}
|
||||
|
||||
/**
|
||||
* Identity a managed workspace instance compares an inbound handoff ticket
|
||||
* against. Every field comes from persisted configuration or injected runtime
|
||||
* identity — never from request headers — so a spoofed `X-Forwarded-Host` or
|
||||
* Tailscale identity header cannot retarget a ticket. Returns null when this
|
||||
* process was not started as a managed workspace, which leaves the exchange
|
||||
* endpoint unregistered.
|
||||
*/
|
||||
export function resolveWorkspaceHandoffIdentity(
|
||||
config: Config,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): WorkspaceHandoffExpectedIdentity | null {
|
||||
const key = resolveWorkspaceHandoffLocalKey(env);
|
||||
if (!key) return null;
|
||||
const configuredOrigin =
|
||||
normalizeWorkspaceHandoffOrigin(env.PAPERCLIP_PUBLIC_URL)
|
||||
?? (config.authBaseUrlMode === "explicit"
|
||||
? normalizeWorkspaceHandoffOrigin(config.authPublicBaseUrl)
|
||||
: null);
|
||||
return {
|
||||
key,
|
||||
instanceId: resolvePaperclipInstanceId(),
|
||||
executionWorkspaceId: resolveWorkspaceHandoffLocalWorkspaceId(env),
|
||||
companyId: resolveWorkspaceHandoffLocalCompanyId(env),
|
||||
origin: configuredOrigin,
|
||||
};
|
||||
}
|
||||
|
||||
export function createBetterAuthInstance(db: Db, config: Config, trustedOrigins: string[]): BetterAuthInstance {
|
||||
const baseUrl = config.authBaseUrlMode === "explicit" ? config.authPublicBaseUrl : undefined;
|
||||
const publicUrl = process.env.PAPERCLIP_PUBLIC_URL?.trim() || baseUrl;
|
||||
@@ -186,6 +224,28 @@ export function createBetterAuthInstance(db: Db, config: Config, trustedOrigins:
|
||||
override: process.env.PAPERCLIP_AUTH_RATE_LIMIT_ENABLED,
|
||||
}),
|
||||
advanced: buildBetterAuthAdvancedOptions({ disableSecureCookies }),
|
||||
// Registered only for a managed workspace instance: the plugin is what makes
|
||||
// `Open workspace` password-independent, and a control-plane instance that
|
||||
// was never handed a workspace key must not expose the exchange at all.
|
||||
...(resolveWorkspaceHandoffIdentity(config)
|
||||
? {
|
||||
plugins: [
|
||||
workspaceLoginHandoffPlugin({
|
||||
db,
|
||||
// Re-resolved per exchange so a hot restart cannot keep validating
|
||||
// against an origin the control plane has since republished.
|
||||
resolveExpectedIdentity: () =>
|
||||
resolveWorkspaceHandoffIdentity(config) ?? {
|
||||
key: null,
|
||||
instanceId: null,
|
||||
executionWorkspaceId: null,
|
||||
companyId: null,
|
||||
origin: null,
|
||||
},
|
||||
}),
|
||||
],
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
|
||||
if (!baseUrl) {
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
/**
|
||||
* Ticket exchange, isolated from Better Auth and Express so the whole
|
||||
* security matrix (replay, expiry, wrong user/origin/workspace/instance,
|
||||
* missing membership, spoofed forwarded headers) is unit-testable.
|
||||
*
|
||||
* Every dependency is injected. The caller supplies expected identity from
|
||||
* *configuration*; this module never reads request headers, which is what makes
|
||||
* forwarded-header spoofing structurally impossible rather than merely checked.
|
||||
*/
|
||||
|
||||
import {
|
||||
sanitizeWorkspaceHandoffRedirectPath,
|
||||
verifyWorkspaceHandoffTicket,
|
||||
type WorkspaceHandoffTicketPayload,
|
||||
type WorkspaceHandoffVerificationFailure,
|
||||
} from "./workspace-login-handoff.js";
|
||||
|
||||
export type WorkspaceHandoffClonedIdentity = {
|
||||
userId: string;
|
||||
email: string | null;
|
||||
name?: string | null;
|
||||
/**
|
||||
* Whether the cloned user has an active membership **in the ticket's company**.
|
||||
* An unscoped "has some membership" answer would let the exchange hand out a
|
||||
* session with no access to the board the caller was opening.
|
||||
*/
|
||||
hasActiveMembership: boolean;
|
||||
};
|
||||
|
||||
export type WorkspaceHandoffExchangeDeps = {
|
||||
/** Resolve the cloned user by the id the ticket names, scoped to its company. */
|
||||
findClonedIdentity: (
|
||||
input: { userId: string; companyId: string },
|
||||
) => Promise<WorkspaceHandoffClonedIdentity | null>;
|
||||
/**
|
||||
* Record the nonce, returning false when it was already recorded.
|
||||
* Must be atomic: this is the only defense against a same-window replay.
|
||||
*/
|
||||
reserveNonce: (input: { nonce: string; expiresAt: Date }) => Promise<boolean>;
|
||||
/** Create an instance-scoped session for the verified cloned user. */
|
||||
createSession: (userId: string) => Promise<void>;
|
||||
};
|
||||
|
||||
export type WorkspaceHandoffExchangeFailureReason =
|
||||
| WorkspaceHandoffVerificationFailure
|
||||
| "replayed"
|
||||
| "unknown_user"
|
||||
| "user_mismatch"
|
||||
| "missing_membership"
|
||||
| "session_failed";
|
||||
|
||||
export type WorkspaceHandoffExchangeResult =
|
||||
| { ok: true; redirectTo: string; payload: WorkspaceHandoffTicketPayload }
|
||||
| { ok: false; reason: WorkspaceHandoffExchangeFailureReason; payload: WorkspaceHandoffTicketPayload | null };
|
||||
|
||||
/**
|
||||
* HTTP status for a failed exchange.
|
||||
*
|
||||
* `not_configured` is the only server-fault case (the guest was started without
|
||||
* a key); everything else is a rejected credential and answers 401 so an
|
||||
* attacker cannot distinguish "wrong workspace" from "expired" by status alone.
|
||||
*/
|
||||
export function workspaceHandoffFailureStatus(reason: WorkspaceHandoffExchangeFailureReason): number {
|
||||
if (reason === "not_configured") return 503;
|
||||
if (reason === "session_failed") return 500;
|
||||
return 401;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a failure means the workspace itself needs repair rather than the
|
||||
* caller needing a fresh ticket. The UI turns these into an explicit
|
||||
* "clone is incomplete" state instead of a generic sign-in error.
|
||||
*/
|
||||
export function isWorkspaceHandoffRepairReason(reason: WorkspaceHandoffExchangeFailureReason): boolean {
|
||||
return reason === "unknown_user" || reason === "missing_membership" || reason === "user_mismatch";
|
||||
}
|
||||
|
||||
export async function exchangeWorkspaceHandoffTicket(input: {
|
||||
ticket: string | null | undefined;
|
||||
key: string | null | undefined;
|
||||
expected: {
|
||||
instanceId: string | null;
|
||||
executionWorkspaceId: string | null;
|
||||
companyId: string | null;
|
||||
origin: string | null;
|
||||
};
|
||||
deps: WorkspaceHandoffExchangeDeps;
|
||||
now?: Date;
|
||||
}): Promise<WorkspaceHandoffExchangeResult> {
|
||||
const verification = verifyWorkspaceHandoffTicket({
|
||||
ticket: input.ticket,
|
||||
key: input.key,
|
||||
expected: input.expected,
|
||||
now: input.now,
|
||||
});
|
||||
if (!verification.ok) return { ok: false, reason: verification.reason, payload: null };
|
||||
const payload = verification.payload;
|
||||
|
||||
const identity = await input.deps.findClonedIdentity({ userId: payload.sub, companyId: payload.cid });
|
||||
if (!identity) return { ok: false, reason: "unknown_user", payload };
|
||||
// The email is bound into the signature, so a mismatch means the clone drifted
|
||||
// from the source instance (a reused id after a reseed), not a forged ticket.
|
||||
if ((identity.email ?? "").trim().toLowerCase() !== payload.email.trim().toLowerCase()) {
|
||||
return { ok: false, reason: "user_mismatch", payload };
|
||||
}
|
||||
if (!identity.hasActiveMembership) return { ok: false, reason: "missing_membership", payload };
|
||||
|
||||
// Reserve last: an identity failure must stay retryable after a repair, while a
|
||||
// ticket that got far enough to mint a session is burned even if that mint fails.
|
||||
const reserved = await input.deps.reserveNonce({
|
||||
nonce: payload.jti,
|
||||
expiresAt: new Date(payload.exp * 1000),
|
||||
});
|
||||
if (!reserved) return { ok: false, reason: "replayed", payload };
|
||||
|
||||
try {
|
||||
await input.deps.createSession(identity.userId);
|
||||
} catch {
|
||||
return { ok: false, reason: "session_failed", payload };
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
redirectTo: sanitizeWorkspaceHandoffRedirectPath(payload.next),
|
||||
payload,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
/**
|
||||
* Better Auth plugin that exchanges a signed workspace login handoff ticket for
|
||||
* an instance-scoped session (PAP-17572).
|
||||
*
|
||||
* Implemented as a plugin endpoint rather than a hand-rolled Express route so
|
||||
* session creation and cookie signing go through Better Auth's own supported
|
||||
* path (`internalAdapter.createSession` + `setSessionCookie`) instead of this
|
||||
* repository reimplementing cookie signing.
|
||||
*
|
||||
* The endpoint is only registered when the process was handed a per-workspace
|
||||
* verification key, so a normal control-plane instance exposes no extra surface.
|
||||
*/
|
||||
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { setSessionCookie } from "better-auth/cookies";
|
||||
import { createAuthEndpoint } from "better-auth/api";
|
||||
import type { Session, User } from "better-auth/types";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { companyMemberships } from "@paperclipai/db";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import {
|
||||
exchangeWorkspaceHandoffTicket,
|
||||
isWorkspaceHandoffRepairReason,
|
||||
type WorkspaceHandoffExchangeFailureReason,
|
||||
} from "./workspace-login-handoff-exchange.js";
|
||||
import {
|
||||
WORKSPACE_HANDOFF_EXCHANGE_PATH,
|
||||
WORKSPACE_HANDOFF_TICKET_QUERY_PARAM,
|
||||
workspaceHandoffKeyFingerprint,
|
||||
} from "./workspace-login-handoff.js";
|
||||
|
||||
/**
|
||||
* Identity the guest expects, resolved from persisted configuration.
|
||||
*
|
||||
* Deliberately a callback: it is re-read per exchange so a hot-restarted guest
|
||||
* cannot keep validating against a stale origin, and it keeps request headers
|
||||
* out of the comparison entirely.
|
||||
*/
|
||||
export type WorkspaceHandoffExpectedIdentity = {
|
||||
key: string | null;
|
||||
instanceId: string | null;
|
||||
executionWorkspaceId: string | null;
|
||||
companyId: string | null;
|
||||
origin: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* The slice of Better Auth's endpoint context this exchange uses.
|
||||
*
|
||||
* Declared structurally rather than importing the library's deeply generic
|
||||
* `GenericEndpointContext`, whose inference depends on the plugin's own option
|
||||
* types. Narrowing here keeps the handler body fully type-checked; the single
|
||||
* cast at the top of the handler is the only unchecked step.
|
||||
*/
|
||||
type WorkspaceHandoffEndpointContext = {
|
||||
query?: Record<string, unknown>;
|
||||
setHeader: (name: string, value: string) => void;
|
||||
redirect: (url: string) => unknown;
|
||||
error: (status: string, body?: Record<string, unknown>) => unknown;
|
||||
context: {
|
||||
internalAdapter: {
|
||||
findUserById: (userId: string) => Promise<User | null>;
|
||||
createSession: (userId: string) => Promise<Session | null>;
|
||||
findVerificationValue: (identifier: string) => Promise<{ identifier: string } | null>;
|
||||
reserveVerificationValue: (data: {
|
||||
identifier: string;
|
||||
value: string;
|
||||
expiresAt: Date;
|
||||
}) => Promise<boolean>;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
/** Where a rejected ticket lands the browser: the labeled snapshot-local fallback. */
|
||||
function buildFallbackRedirect(input: {
|
||||
baseUrl: string | undefined;
|
||||
next: string;
|
||||
reason: WorkspaceHandoffExchangeFailureReason;
|
||||
}): string {
|
||||
const target = new URL("/auth", input.baseUrl ?? "http://localhost");
|
||||
target.searchParams.set("next", input.next);
|
||||
target.searchParams.set("workspaceHandoffError", input.reason);
|
||||
return input.baseUrl ? target.toString() : `${target.pathname}${target.search}`;
|
||||
}
|
||||
|
||||
export function workspaceLoginHandoffPlugin(deps: {
|
||||
db: Db;
|
||||
resolveExpectedIdentity: () => WorkspaceHandoffExpectedIdentity;
|
||||
}) {
|
||||
return {
|
||||
id: "paperclip-workspace-login-handoff",
|
||||
endpoints: {
|
||||
exchangeWorkspaceLoginHandoff: createAuthEndpoint(
|
||||
WORKSPACE_HANDOFF_EXCHANGE_PATH,
|
||||
{ method: "GET", requireHeaders: true },
|
||||
async (endpointContext) => {
|
||||
const ctx = endpointContext as unknown as WorkspaceHandoffEndpointContext;
|
||||
const expected = deps.resolveExpectedIdentity();
|
||||
const rawTicket = ctx.query?.[WORKSPACE_HANDOFF_TICKET_QUERY_PARAM];
|
||||
const ticket = typeof rawTicket === "string" ? rawTicket : null;
|
||||
|
||||
// `no-store` keeps the ticket-bearing response out of shared caches and
|
||||
// `no-referrer` stops the ticket URL from leaking to the landing page.
|
||||
ctx.setHeader("Cache-Control", "no-store");
|
||||
ctx.setHeader("Referrer-Policy", "no-referrer");
|
||||
|
||||
const result = await exchangeWorkspaceHandoffTicket({
|
||||
ticket,
|
||||
key: expected.key,
|
||||
expected: {
|
||||
instanceId: expected.instanceId,
|
||||
executionWorkspaceId: expected.executionWorkspaceId,
|
||||
companyId: expected.companyId,
|
||||
origin: expected.origin,
|
||||
},
|
||||
deps: {
|
||||
findClonedIdentity: async ({ userId, companyId }) => {
|
||||
const user = await ctx.context.internalAdapter.findUserById(userId);
|
||||
if (!user) return null;
|
||||
// Membership lives in Paperclip's own schema, so it is read
|
||||
// through the app's `db` handle rather than the Better Auth
|
||||
// adapter. Scoped to the ticket's company: an unscoped check
|
||||
// would accept a clone where this user belongs to some *other*
|
||||
// company and hand them a session with no access to the board
|
||||
// they opened. Existence, not a count.
|
||||
const activeMemberships = await deps.db
|
||||
.select({ id: companyMemberships.id })
|
||||
.from(companyMemberships)
|
||||
.where(
|
||||
and(
|
||||
eq(companyMemberships.companyId, companyId),
|
||||
eq(companyMemberships.principalType, "user"),
|
||||
eq(companyMemberships.principalId, userId),
|
||||
eq(companyMemberships.status, "active"),
|
||||
),
|
||||
)
|
||||
.limit(1)
|
||||
.then((rows) => rows.length);
|
||||
return {
|
||||
userId: user.id,
|
||||
email: user.email ?? null,
|
||||
name: user.name ?? null,
|
||||
hasActiveMembership: activeMemberships > 0,
|
||||
};
|
||||
},
|
||||
reserveNonce: async ({ nonce, expiresAt }) => {
|
||||
const identifier = `workspace-login-handoff:${nonce}`;
|
||||
// Two layers, because each covers a case the other misses. The
|
||||
// lookup catches an ordinary sequential replay on any storage
|
||||
// backend; the reservation is a primary-key insert, so it is the
|
||||
// one that survives two exchanges racing inside the TTL. Relying
|
||||
// on the reservation alone would silently depend on the
|
||||
// configured adapter enforcing id uniqueness.
|
||||
if (await ctx.context.internalAdapter.findVerificationValue(identifier)) return false;
|
||||
return await ctx.context.internalAdapter.reserveVerificationValue({
|
||||
identifier,
|
||||
value: "consumed",
|
||||
// Keep the record slightly past the ticket's own expiry so a
|
||||
// replay attempted at the very edge of the window still loses.
|
||||
expiresAt: new Date(expiresAt.getTime() + 60_000),
|
||||
});
|
||||
},
|
||||
createSession: async (userId) => {
|
||||
const user = await ctx.context.internalAdapter.findUserById(userId);
|
||||
if (!user) throw new Error("cloned user disappeared between verification and session creation");
|
||||
const session = await ctx.context.internalAdapter.createSession(userId);
|
||||
if (!session) throw new Error("Better Auth did not return a session");
|
||||
await setSessionCookie(ctx as never, { session, user });
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
if (!result.ok) {
|
||||
// Audit with the nonce and reason but never the ticket itself.
|
||||
logger.warn(
|
||||
{
|
||||
reason: result.reason,
|
||||
nonce: result.payload?.jti ?? null,
|
||||
executionWorkspaceId: expected.executionWorkspaceId,
|
||||
instanceId: expected.instanceId,
|
||||
keyFingerprint: expected.key ? workspaceHandoffKeyFingerprint(expected.key) : null,
|
||||
needsRepair: isWorkspaceHandoffRepairReason(result.reason),
|
||||
},
|
||||
"workspace login handoff rejected",
|
||||
);
|
||||
if (result.reason === "not_configured" || result.reason === "session_failed") {
|
||||
throw ctx.error(result.reason === "not_configured" ? "SERVICE_UNAVAILABLE" : "INTERNAL_SERVER_ERROR", {
|
||||
message: "Workspace login handoff is unavailable",
|
||||
code: result.reason,
|
||||
});
|
||||
}
|
||||
throw ctx.redirect(
|
||||
buildFallbackRedirect({
|
||||
baseUrl: expected.origin ?? undefined,
|
||||
next: result.payload?.next ?? "/",
|
||||
reason: result.reason,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
logger.info(
|
||||
{
|
||||
nonce: result.payload.jti,
|
||||
userId: result.payload.sub,
|
||||
executionWorkspaceId: expected.executionWorkspaceId,
|
||||
instanceId: expected.instanceId,
|
||||
issuerInstanceId: result.payload.iss,
|
||||
},
|
||||
"workspace login handoff accepted",
|
||||
);
|
||||
// An HTTP redirect (not a client-side navigation) is what keeps the
|
||||
// ticket URL out of the browser's session history.
|
||||
throw ctx.redirect(
|
||||
expected.origin
|
||||
? new URL(result.redirectTo, expected.origin).toString()
|
||||
: result.redirectTo,
|
||||
);
|
||||
},
|
||||
),
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,406 @@
|
||||
/**
|
||||
* Signed workspace login handoff (PAP-17572).
|
||||
*
|
||||
* A board user opening a managed workspace must not have to know which cloned
|
||||
* password is current. The authenticated main control plane mints a short-lived,
|
||||
* single-use ticket; the isolated workspace verifies it against its own copy of
|
||||
* the signing key and creates its own instance-scoped session.
|
||||
*
|
||||
* Threat model notes that drive the shape below:
|
||||
*
|
||||
* - **Nothing browser-supplied is trusted.** The audience is the origin the
|
||||
* control plane published for that runtime and the workspace compares it
|
||||
* against its *configured* public origin. `Host`, `X-Forwarded-Host`,
|
||||
* `X-Forwarded-Proto` and Tailscale identity headers are never consulted, so
|
||||
* a spoofed forwarded header cannot retarget or authorize a ticket.
|
||||
* - **Keys are per workspace.** The control plane keeps the root secret and
|
||||
* hands each guest only `HMAC(root, instanceId|executionWorkspaceId)`. A
|
||||
* compromised workspace therefore cannot mint a ticket for a sibling
|
||||
* workspace, and the guest never sees material that would let it do so.
|
||||
* - **Replay is bounded twice.** A short expiry limits the window and the
|
||||
* workspace records the nonce on first use, so a captured ticket cannot be
|
||||
* exchanged a second time even inside that window.
|
||||
*/
|
||||
|
||||
import { createHmac, randomBytes, timingSafeEqual } from "node:crypto";
|
||||
|
||||
/** Ticket envelope version. Bumped whenever the signed payload shape changes. */
|
||||
export const WORKSPACE_HANDOFF_TICKET_VERSION = "wh1";
|
||||
|
||||
/**
|
||||
* Default ticket lifetime. Long enough for a click plus a cold guest response,
|
||||
* short enough that a ticket leaked through a proxy log is worthless.
|
||||
*/
|
||||
export const WORKSPACE_HANDOFF_TICKET_TTL_SECONDS = 90;
|
||||
|
||||
/** Tolerated clock skew between the control plane and the isolated workspace. */
|
||||
export const WORKSPACE_HANDOFF_CLOCK_SKEW_SECONDS = 30;
|
||||
|
||||
/** Path the guest exposes for the exchange, relative to the Better Auth mount. */
|
||||
export const WORKSPACE_HANDOFF_EXCHANGE_PATH = "/workspace-handoff/exchange";
|
||||
|
||||
/** Query parameter carrying the ticket. Redacted from request logs by name. */
|
||||
export const WORKSPACE_HANDOFF_TICKET_QUERY_PARAM = "ticket";
|
||||
|
||||
const HANDOFF_KEY_ENV = "PAPERCLIP_WORKSPACE_HANDOFF_KEY";
|
||||
const HANDOFF_ROOT_SECRET_ENV = "PAPERCLIP_WORKSPACE_HANDOFF_SECRET";
|
||||
const READINESS_TOKEN_ENV = "PAPERCLIP_WORKSPACE_READINESS_TOKEN";
|
||||
const EXECUTION_WORKSPACE_ID_ENV = "PAPERCLIP_EXECUTION_WORKSPACE_ID";
|
||||
const EXECUTION_WORKSPACE_COMPANY_ID_ENV = "PAPERCLIP_EXECUTION_WORKSPACE_COMPANY_ID";
|
||||
|
||||
export const WORKSPACE_HANDOFF_KEY_ENV_KEY = HANDOFF_KEY_ENV;
|
||||
export const WORKSPACE_READINESS_TOKEN_ENV_KEY = READINESS_TOKEN_ENV;
|
||||
export const WORKSPACE_EXECUTION_WORKSPACE_ID_ENV_KEY = EXECUTION_WORKSPACE_ID_ENV;
|
||||
export const WORKSPACE_EXECUTION_WORKSPACE_COMPANY_ID_ENV_KEY = EXECUTION_WORKSPACE_COMPANY_ID_ENV;
|
||||
export const WORKSPACE_READINESS_TOKEN_HEADER = "x-paperclip-workspace-readiness-token";
|
||||
export const WORKSPACE_READINESS_USER_ID_HEADER = "x-paperclip-workspace-readiness-user-id";
|
||||
export const WORKSPACE_READINESS_USER_EMAIL_HEADER = "x-paperclip-workspace-readiness-user-email";
|
||||
|
||||
export type WorkspaceHandoffTicketPayload = {
|
||||
/** Envelope version. */
|
||||
v: typeof WORKSPACE_HANDOFF_TICKET_VERSION;
|
||||
/** Single-use nonce; the guest records it on first exchange. */
|
||||
jti: string;
|
||||
/** Cloned user id the guest must sign in. */
|
||||
sub: string;
|
||||
/** Expected email of that cloned user, compared case-insensitively. */
|
||||
email: string;
|
||||
/** Execution workspace the ticket is scoped to. */
|
||||
ws: string;
|
||||
/**
|
||||
* Company whose board this workspace represents.
|
||||
*
|
||||
* Bound because "the cloned user has *some* active membership" is not the
|
||||
* question worth asking: a multi-company clone can satisfy it while leaving the
|
||||
* signed-in user with no access to the company they were opening, which lands
|
||||
* them on an empty board with no explanation.
|
||||
*/
|
||||
cid: string;
|
||||
/** Isolated instance id the ticket is scoped to. */
|
||||
iid: string;
|
||||
/** Origin the control plane published for this runtime. */
|
||||
aud: string;
|
||||
/** Issuing instance id, recorded for audit. */
|
||||
iss: string;
|
||||
/** Seconds since epoch. */
|
||||
iat: number;
|
||||
exp: number;
|
||||
/** Relative path to land on after the exchange. */
|
||||
next: string;
|
||||
};
|
||||
|
||||
export type WorkspaceHandoffVerificationFailure =
|
||||
| "not_configured"
|
||||
| "malformed"
|
||||
| "unsupported_version"
|
||||
| "bad_signature"
|
||||
| "expired"
|
||||
| "not_yet_valid"
|
||||
| "origin_mismatch"
|
||||
| "instance_mismatch"
|
||||
| "workspace_mismatch"
|
||||
| "company_mismatch";
|
||||
|
||||
export type WorkspaceHandoffVerificationResult =
|
||||
| { ok: true; payload: WorkspaceHandoffTicketPayload }
|
||||
| { ok: false; reason: WorkspaceHandoffVerificationFailure };
|
||||
|
||||
function base64UrlEncode(value: Buffer): string {
|
||||
return value.toString("base64url");
|
||||
}
|
||||
|
||||
function base64UrlDecode(value: string): Buffer | null {
|
||||
// `Buffer.from` silently drops invalid characters, so round-trip to prove the
|
||||
// input really was canonical base64url before trusting the bytes.
|
||||
if (!/^[A-Za-z0-9_-]+$/.test(value)) return null;
|
||||
const decoded = Buffer.from(value, "base64url");
|
||||
if (decoded.length === 0) return null;
|
||||
return base64UrlEncode(decoded) === value ? decoded : null;
|
||||
}
|
||||
|
||||
function sign(key: string, message: string): Buffer {
|
||||
return createHmac("sha256", key).update(message).digest();
|
||||
}
|
||||
|
||||
function constantTimeMatches(expected: Buffer, provided: Buffer): boolean {
|
||||
if (expected.length !== provided.length) return false;
|
||||
return timingSafeEqual(expected, provided);
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize an origin for comparison: scheme + host + explicit non-default port.
|
||||
* Returns null when the input is not an absolute http(s) URL, so an unparseable
|
||||
* or relative value can never accidentally compare equal.
|
||||
*/
|
||||
export function normalizeWorkspaceHandoffOrigin(value: string | null | undefined): string | null {
|
||||
const trimmed = value?.trim();
|
||||
if (!trimmed) return null;
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(trimmed);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (parsed.protocol !== "https:" && parsed.protocol !== "http:") return null;
|
||||
return parsed.origin;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reduce a caller-supplied landing target to a safe same-origin path.
|
||||
*
|
||||
* Anything absolute, protocol-relative, or backslash-escaped collapses to `/`.
|
||||
* A handoff must never double as an open redirect.
|
||||
*/
|
||||
export function sanitizeWorkspaceHandoffRedirectPath(value: string | null | undefined): string {
|
||||
const trimmed = value?.trim();
|
||||
if (!trimmed) return "/";
|
||||
if (!trimmed.startsWith("/")) return "/";
|
||||
// `//host` and `/\host` are both browser-recognized protocol-relative forms.
|
||||
if (trimmed.startsWith("//") || trimmed.startsWith("/\\")) return "/";
|
||||
// Control characters and raw whitespace can split a `Location` header.
|
||||
if (/[\u0000-\u0020\u007f]/.test(trimmed)) return "/";
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the per-workspace signing key handed to one guest instance.
|
||||
*
|
||||
* Domain-separated from every other use of the root secret and bound to both
|
||||
* identities, so a key only ever validates tickets for the exact workspace it
|
||||
* was issued to.
|
||||
*/
|
||||
export function deriveWorkspaceHandoffKey(input: {
|
||||
rootSecret: string;
|
||||
instanceId: string;
|
||||
executionWorkspaceId: string;
|
||||
}): string {
|
||||
return createHmac("sha256", input.rootSecret)
|
||||
.update(`paperclip.workspace-login-handoff.${WORKSPACE_HANDOFF_TICKET_VERSION}\n`)
|
||||
.update(`${input.instanceId}\n`)
|
||||
.update(`${input.executionWorkspaceId}`)
|
||||
.digest("hex");
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the bearer token the control plane presents to read a guest's
|
||||
* protected readiness. Separate derivation from {@link deriveWorkspaceHandoffKey}
|
||||
* so a probe token that leaks through an HTTP client log cannot sign tickets.
|
||||
*/
|
||||
export function deriveWorkspaceReadinessToken(input: {
|
||||
rootSecret: string;
|
||||
instanceId: string;
|
||||
executionWorkspaceId: string;
|
||||
}): string {
|
||||
return createHmac("sha256", input.rootSecret)
|
||||
.update(`paperclip.workspace-readiness-probe.${WORKSPACE_HANDOFF_TICKET_VERSION}\n`)
|
||||
.update(`${input.instanceId}\n`)
|
||||
.update(`${input.executionWorkspaceId}`)
|
||||
.digest("hex");
|
||||
}
|
||||
|
||||
/**
|
||||
* Root secret for handoff key derivation, on the control-plane side only.
|
||||
*
|
||||
* A dedicated `PAPERCLIP_WORKSPACE_HANDOFF_SECRET` is preferred. When it is
|
||||
* absent we derive dedicated key material from the instance's existing signing
|
||||
* secret rather than disabling the feature: the alternative is that every
|
||||
* already-deployed instance silently falls back to "remember the cloned
|
||||
* password", which is the defect this work exists to remove. The derivation is
|
||||
* one-way and domain-separated, so the handoff key is never equal to — and
|
||||
* cannot be used to recover — the secret it came from.
|
||||
*/
|
||||
export function resolveWorkspaceHandoffRootSecret(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): { secret: string; source: "dedicated" | "derived" } | null {
|
||||
const dedicated = env[HANDOFF_ROOT_SECRET_ENV]?.trim();
|
||||
if (dedicated) return { secret: dedicated, source: "dedicated" };
|
||||
|
||||
const fallback = env.BETTER_AUTH_SECRET?.trim() || env.PAPERCLIP_AGENT_JWT_SECRET?.trim();
|
||||
if (!fallback) return null;
|
||||
return {
|
||||
secret: createHmac("sha256", fallback)
|
||||
.update(`paperclip.workspace-login-handoff.root.${WORKSPACE_HANDOFF_TICKET_VERSION}`)
|
||||
.digest("hex"),
|
||||
source: "derived",
|
||||
};
|
||||
}
|
||||
|
||||
/** The verification key injected into this (guest) process, when present. */
|
||||
export function resolveWorkspaceHandoffLocalKey(env: NodeJS.ProcessEnv = process.env): string | null {
|
||||
return env[HANDOFF_KEY_ENV]?.trim() || null;
|
||||
}
|
||||
|
||||
/** The readiness-probe token injected into this (guest) process, when present. */
|
||||
export function resolveWorkspaceReadinessLocalToken(env: NodeJS.ProcessEnv = process.env): string | null {
|
||||
return env[READINESS_TOKEN_ENV]?.trim() || null;
|
||||
}
|
||||
|
||||
/** The execution workspace this (guest) process was provisioned for. */
|
||||
export function resolveWorkspaceHandoffLocalWorkspaceId(env: NodeJS.ProcessEnv = process.env): string | null {
|
||||
return env[EXECUTION_WORKSPACE_ID_ENV]?.trim() || null;
|
||||
}
|
||||
|
||||
/** The company whose board this (guest) workspace represents. */
|
||||
export function resolveWorkspaceHandoffLocalCompanyId(env: NodeJS.ProcessEnv = process.env): string | null {
|
||||
return env[EXECUTION_WORKSPACE_COMPANY_ID_ENV]?.trim() || null;
|
||||
}
|
||||
|
||||
export function issueWorkspaceHandoffTicket(input: {
|
||||
key: string;
|
||||
userId: string;
|
||||
email: string;
|
||||
executionWorkspaceId: string;
|
||||
companyId: string;
|
||||
instanceId: string;
|
||||
origin: string;
|
||||
issuerInstanceId: string;
|
||||
next?: string | null;
|
||||
ttlSeconds?: number;
|
||||
now?: Date;
|
||||
nonce?: string;
|
||||
}): { ticket: string; payload: WorkspaceHandoffTicketPayload } {
|
||||
const origin = normalizeWorkspaceHandoffOrigin(input.origin);
|
||||
if (!origin) {
|
||||
throw new Error("Workspace login handoff needs an absolute http(s) target origin");
|
||||
}
|
||||
const issuedAtMs = (input.now ?? new Date()).getTime();
|
||||
const issuedAt = Math.floor(issuedAtMs / 1000);
|
||||
const ttlSeconds = Math.max(1, Math.floor(input.ttlSeconds ?? WORKSPACE_HANDOFF_TICKET_TTL_SECONDS));
|
||||
const payload: WorkspaceHandoffTicketPayload = {
|
||||
v: WORKSPACE_HANDOFF_TICKET_VERSION,
|
||||
jti: input.nonce ?? randomBytes(18).toString("base64url"),
|
||||
sub: input.userId,
|
||||
email: input.email,
|
||||
ws: input.executionWorkspaceId,
|
||||
cid: input.companyId,
|
||||
iid: input.instanceId,
|
||||
aud: origin,
|
||||
iss: input.issuerInstanceId,
|
||||
iat: issuedAt,
|
||||
exp: issuedAt + ttlSeconds,
|
||||
next: sanitizeWorkspaceHandoffRedirectPath(input.next),
|
||||
};
|
||||
const encodedPayload = base64UrlEncode(Buffer.from(JSON.stringify(payload), "utf8"));
|
||||
const signedMessage = `${WORKSPACE_HANDOFF_TICKET_VERSION}.${encodedPayload}`;
|
||||
const signature = base64UrlEncode(sign(input.key, signedMessage));
|
||||
return { ticket: `${signedMessage}.${signature}`, payload };
|
||||
}
|
||||
|
||||
/** Absolute exchange URL for a minted ticket. */
|
||||
export function buildWorkspaceHandoffExchangeUrl(input: { origin: string; ticket: string }): string {
|
||||
const origin = normalizeWorkspaceHandoffOrigin(input.origin);
|
||||
if (!origin) throw new Error("Workspace login handoff needs an absolute http(s) target origin");
|
||||
const url = new URL(`/api/auth${WORKSPACE_HANDOFF_EXCHANGE_PATH}`, origin);
|
||||
url.searchParams.set(WORKSPACE_HANDOFF_TICKET_QUERY_PARAM, input.ticket);
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function isPayloadShape(value: unknown): value is WorkspaceHandoffTicketPayload {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const candidate = value as Record<string, unknown>;
|
||||
return (
|
||||
typeof candidate.jti === "string" && candidate.jti.length > 0
|
||||
&& typeof candidate.sub === "string" && candidate.sub.length > 0
|
||||
&& typeof candidate.email === "string" && candidate.email.length > 0
|
||||
&& typeof candidate.ws === "string" && candidate.ws.length > 0
|
||||
&& typeof candidate.cid === "string" && candidate.cid.length > 0
|
||||
&& typeof candidate.iid === "string" && candidate.iid.length > 0
|
||||
&& typeof candidate.aud === "string" && candidate.aud.length > 0
|
||||
&& typeof candidate.iss === "string"
|
||||
&& Number.isFinite(candidate.iat)
|
||||
&& Number.isFinite(candidate.exp)
|
||||
&& typeof candidate.next === "string"
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify a ticket against this workspace's own key and identity.
|
||||
*
|
||||
* `expected` must come from persisted configuration. Passing header-derived
|
||||
* values would reintroduce exactly the spoofing path this design closes.
|
||||
*/
|
||||
export function verifyWorkspaceHandoffTicket(input: {
|
||||
ticket: string | null | undefined;
|
||||
key: string | null | undefined;
|
||||
expected: {
|
||||
instanceId: string | null;
|
||||
executionWorkspaceId: string | null;
|
||||
companyId: string | null;
|
||||
origin: string | null;
|
||||
};
|
||||
now?: Date;
|
||||
clockSkewSeconds?: number;
|
||||
}): WorkspaceHandoffVerificationResult {
|
||||
const key = input.key?.trim();
|
||||
if (!key) return { ok: false, reason: "not_configured" };
|
||||
|
||||
const expectedOrigin = normalizeWorkspaceHandoffOrigin(input.expected.origin);
|
||||
const expectedInstanceId = input.expected.instanceId?.trim();
|
||||
const expectedWorkspaceId = input.expected.executionWorkspaceId?.trim();
|
||||
const expectedCompanyId = input.expected.companyId?.trim();
|
||||
// Fail closed on unresolved local identity: without it the checks below would
|
||||
// degrade into "any validly signed ticket wins".
|
||||
if (!expectedOrigin) return { ok: false, reason: "origin_mismatch" };
|
||||
if (!expectedInstanceId) return { ok: false, reason: "instance_mismatch" };
|
||||
if (!expectedWorkspaceId) return { ok: false, reason: "workspace_mismatch" };
|
||||
if (!expectedCompanyId) return { ok: false, reason: "company_mismatch" };
|
||||
|
||||
const ticket = input.ticket?.trim();
|
||||
if (!ticket) return { ok: false, reason: "malformed" };
|
||||
const segments = ticket.split(".");
|
||||
if (segments.length !== 3) return { ok: false, reason: "malformed" };
|
||||
const [version, encodedPayload, encodedSignature] = segments as [string, string, string];
|
||||
if (version !== WORKSPACE_HANDOFF_TICKET_VERSION) return { ok: false, reason: "unsupported_version" };
|
||||
|
||||
const providedSignature = base64UrlDecode(encodedSignature);
|
||||
if (!providedSignature) return { ok: false, reason: "malformed" };
|
||||
const expectedSignature = sign(key, `${version}.${encodedPayload}`);
|
||||
// Signature first: never parse attacker-controlled JSON we have not authenticated.
|
||||
if (!constantTimeMatches(expectedSignature, providedSignature)) return { ok: false, reason: "bad_signature" };
|
||||
|
||||
const decodedPayload = base64UrlDecode(encodedPayload);
|
||||
if (!decodedPayload) return { ok: false, reason: "malformed" };
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(decodedPayload.toString("utf8"));
|
||||
} catch {
|
||||
return { ok: false, reason: "malformed" };
|
||||
}
|
||||
if (!isPayloadShape(parsed)) return { ok: false, reason: "malformed" };
|
||||
if (parsed.v !== WORKSPACE_HANDOFF_TICKET_VERSION) return { ok: false, reason: "unsupported_version" };
|
||||
|
||||
const nowSeconds = Math.floor((input.now ?? new Date()).getTime() / 1000);
|
||||
const skew = Math.max(0, Math.floor(input.clockSkewSeconds ?? WORKSPACE_HANDOFF_CLOCK_SKEW_SECONDS));
|
||||
if (parsed.exp + skew < nowSeconds) return { ok: false, reason: "expired" };
|
||||
if (parsed.iat - skew > nowSeconds) return { ok: false, reason: "not_yet_valid" };
|
||||
if (parsed.exp <= parsed.iat) return { ok: false, reason: "malformed" };
|
||||
|
||||
if (normalizeWorkspaceHandoffOrigin(parsed.aud) !== expectedOrigin) {
|
||||
return { ok: false, reason: "origin_mismatch" };
|
||||
}
|
||||
if (parsed.iid !== expectedInstanceId) return { ok: false, reason: "instance_mismatch" };
|
||||
if (parsed.ws !== expectedWorkspaceId) return { ok: false, reason: "workspace_mismatch" };
|
||||
if (parsed.cid !== expectedCompanyId) return { ok: false, reason: "company_mismatch" };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
payload: { ...parsed, next: sanitizeWorkspaceHandoffRedirectPath(parsed.next) },
|
||||
};
|
||||
}
|
||||
|
||||
/** Verification-key identifier safe to log: proves which key, reveals no key bytes. */
|
||||
export function workspaceHandoffKeyFingerprint(key: string): string {
|
||||
return createHmac("sha256", "paperclip.workspace-login-handoff.fingerprint").update(key).digest("hex").slice(0, 12);
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip the ticket from a URL or URL-ish string before it reaches a log sink.
|
||||
* Applied by the request logger and by the guest's own audit records.
|
||||
*/
|
||||
export function redactWorkspaceHandoffTicket(value: string): string {
|
||||
if (!value.includes(WORKSPACE_HANDOFF_TICKET_QUERY_PARAM)) return value;
|
||||
return value.replace(
|
||||
new RegExp(`([?&]${WORKSPACE_HANDOFF_TICKET_QUERY_PARAM}=)[^&#\\s]+`, "gi"),
|
||||
"$1[redacted]",
|
||||
);
|
||||
}
|
||||
Reference in new issue
Block a user