Files
PaperClipAI/server/src/services/public-mcp/client-mirror-sync.ts
T
DottaandClaude Opus 5.5 904c01d1a3 feat(mcp): graduate Assistant connections (MCP) out of experimental (#22015)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assistant connections (MCP) let an outside assistant, or another
Paperclip instance, sign in as a person and use the `/mcp/paperclip`
endpoint
> - The feature was behind the experimental `enablePublicMcp` setting.
The setting was off by default and in practice off on Cloud stacks
> - Instance-to-instance work (one person's agent files a task on a
colleague's instance) fails when the receiving instance has the setting
off
> - This pull request moves the feature out of Experimental. It is now
always on, and a stored or Cloud-managed "off" value is ignored
> - The benefit is that every authenticated instance with a public URL,
including Cloud workspaces, accepts assistant connections with no setup
step

## Linked Issues or Issue Description

**Subsystem affected**
Public MCP (`server/src/services/public-mcp/`), instance settings,
Connectors → Assistant Connection (MCP) setup page, Experimental
settings page.

**Problem or motivation**
A remote MCP connector on one instance cannot use another instance's
`/mcp/paperclip` unless an admin on the other instance finds and turns
on an experimental toggle. On Cloud, the toggle is off and is managed by
the fleet. Instance-to-instance delegation therefore does not work by
default.

**Proposed solution**
Remove the gate. Keep `enablePublicMcp` only as a compatibility key, the
same way `enableApps` was graduated: old stored rows, PATCH bodies and
Cloud managed configs still parse, but the effective value is always
`true`.

**Alternatives considered**
An instance-admin kill switch that defaults ON. I did not keep one. Each
person already controls access: a connection exists only after that
person approves OAuth consent for one company, and the person or an
admin can revoke it. Deployment policy (authenticated mode and a valid
public URL) still decides whether the endpoint exists. A second switch
would add a state that can silently break instance-to-instance work
again. A schema migration was also not needed: the value lives in the
`experimental` JSON column, and the stored key becomes inert.

**Roadmap alignment**
Requested by the maintainer to enable instance-to-instance
communication.

## What Changed

- `packages/shared`: `enablePublicMcp` is a deprecated compatibility
key. The schema default is `true`. The feature catalog entry is
"Assistant connections (compatibility)" with both defaults `true`.
- `server` instance settings: normalization always returns
`enablePublicMcp: true`. The Cloud managed overlay skips the key, so a
fleet value of `false` cannot disable it. Removed the key from the Cloud
staging experimental-key list. Removed the client-mirror policy step
that ran only when the toggle changed.
- `server` public MCP: personal discovery, OAuth, tools, setup and event
delivery no longer read instance settings (`isEnabled()` returns `true`
for the personal endpoint without a DB read). Dot, Grok and Muse agent
connections keep their own experimental settings.
- `server` Cloud paths: the client-mirror sync, the hosted event
authority, the ingress-handoff expiry worker and the idle sleep proof
treat the front door as enabled. The warm-standby `503
workspace_unclaimed` behavior is unchanged.
- `server` Dot/Grok gates: they no longer also require
`enablePublicMcp`. Error messages no longer tell people to enable
Assistant connections.
- `ui`: removed the Experimental toggle, the "Enable assistant
connections / Open Experimental settings" block on the Assistant
Connection page, and the "Enable Assistant connections (MCP)" hint in
Dot setup. Updated the Dot and Grok copy.
- Storybook: deleted the Experimental-setting story and removed the
"enable the experiment" step from the journey and presenter.
- Docs: `doc/public-mcp.md` ("Enable an instance", "Start from
Connections", `GET /api/mcp/setup`), `doc/openai-dot-runner.md`,
`doc/idle-grok-dispatch.md`, and the agent-access map.
- Tests: new red/green tests for "enabled with no setting stored" and "a
stored or managed off value is ignored". Removed the tests for the
"disabled" behavior, which no longer exists.

## Verification

- Red/green: with the old source and the new tests,
`instance-settings-readonly.test.ts`,
`instance-settings-service.test.ts` and `public-mcp.test.ts` fail (4
failures, and the public-MCP suite fails in `beforeAll` because
`isEnabled()` was `false` with no setting stored). With the new source
they pass.
- `server`: `public-mcp.test.ts`, instance-settings (readonly, service,
managed overlay), `managed-config`, `cloud-event-authority`,
`events-work`, `dot-runner`, `dot-oauth-onboarding` and
`agent-runtime-assignments`. 354 passed.
- Cloud client-mirror lifecycle
(`client-mirror-lifecycle.postgres.test.ts`, skipped in CI) passed
against a local Postgres 17 with
`PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL`. 6/6, plus `events-work` 28/28.
- Idle and sleep: `idle-incoming-wake` (service and route),
`idle-deadline-proof`, `idle-incoming-wake-capability`,
`idle-sleep-proof`, `client-mirror-ack`, `recovery-deadline-acceptance`
and `idle-completed-history` pass. The warm-standby test now also
asserts that discovery after the claim does no SQL read.
- `ui`: `InstanceExperimentalSettings`, `AssistantConnection`,
`NewAgentDialog`, `NewAgent` and `ExternalAgentInviteDialog.muse` pass.
267 passed.
- Typecheck: `packages/shared`, `server` and `ui` pass. `node
scripts/check-maps.mjs` passes.
- `pnpm verify`: see the CI checks on this PR.
- After the deploy: confirm
`/.well-known/oauth-protected-resource/mcp/paperclip` and `/mcp/setup`
return 200 on paperclip.staging without changing any setting.

## Risks

- Behavior change: instances that had the setting off now serve
discovery, OAuth and `/mcp/paperclip`. Security invariants are
unchanged: the endpoint is still mounted only in authenticated mode with
a valid public URL, OAuth still needs a real signed-in user's consent,
local implicit board authority cannot approve, and revocation works the
same way.
- Cloud sleep: on a managed stack, the MCP sleep proof blocks sleep
until the client mirror publishes the tenant front door to Cloud. The
client-mirror sync runs at boot and does this once. Stacks that already
had the flag on use this path today. A stack whose Cloud connector
cannot publish the mirror would stay awake. Watch fleet sleep after the
deploy.
- Personal public-MCP requests now read instance settings one time less.
- Fleet admin: the Cloud fleet catalog can still show and set
`enablePublicMcp`. The value is ignored. Removing it from the fleet
catalog is a separate change in the Cloud repo.
- No migration. The `0354` trigger still compares the stored key. That
causes at most one extra connector-runtime invalidation if an old client
writes the key.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:42:36 -06:00

169 lines
14 KiB
TypeScript

import { mcpClientMirrorClientPages, mcpClientMirrorDevicePages } from "./client-mirror-pages.js";
import { getCloudRuntimeIdentity } from "../cloud-runtime-identity.js";
import { mcpDeviceStatusEntrySchema,mcpDeviceStatusFingerprint,mcpDeviceStatusGeneration,mcpDeviceStatusKeyId } from "./device-status-contract.js";
import { deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js";
import { prepareMcpClientMutation } from "./client-mirror-mutations.js";
import { installMcpDeviceCutover } from "./device-cutover.js";
import { prepareMcpClientMirrorPolicy } from "./client-mirror-mutations.js";
import { and, eq, gt, lte, sql } from "drizzle-orm";
import { mcpOauthClients as clients,mcpOauthDeviceRequests as devices, mcpOauthClientMirrorState as state, mcpOauthClientChanges as changes, type Db } from "@paperclipai/db";
import { createPaperclipCloudConnector, paperclipCloudConnectorConfigFromEnv } from "../paperclip-cloud-connector.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { mcpClientMirrorGeneration, mcpClientMirrorReceiptSchema, mcpClientMirrorResponseSchema, mcpMirroredClientSchema,
type McpClientMirrorChange, type McpClientMirrorRequest, type McpClientMirrorReceipt } from "./client-mirror-contract.js";
export function sameMcpClientMirrorReceipt(left:McpClientMirrorReceipt,right:McpClientMirrorReceipt){
return left.version===right.version&&left.instanceId===right.instanceId&&left.origin===right.origin
&&left.revision===right.revision&&left.enabled===right.enabled&&left.generation===right.generation
&&left.frontDoor===right.frontDoor&&(left.authorizationOrigin??left.origin)===(right.authorizationOrigin??right.origin)
&&left.deviceStatus?.keyId===right.deviceStatus?.keyId&&left.deviceStatus?.generation===right.deviceStatus?.generation
&&left.deviceStatus?.count===right.deviceStatus?.count;
}
export function startMcpClientMirrorSync(db: Db, coordinator: Pick<ReturnType<typeof createDeliveryWorkCoordinator>,"register">,
origin: string | undefined, input: {authorizationOrigin?:string;instanceId?:string;publish?:(request:McpClientMirrorRequest)=>Promise<unknown>} = {}) {
const config = paperclipCloudConnectorConfigFromEnv();
const connector = config ? createPaperclipCloudConnector({config}) : undefined;
const instanceId = input.instanceId ?? config?.instanceId;
const publish = input.publish ?? (connector ? (request:McpClientMirrorRequest) => connector.syncMcpClientMirror(JSON.stringify(request)) : undefined);
let pending = true,refreshPublished=true,fullRecovery=false,reserveFullRevision=false;
let boundStackId=getCloudRuntimeIdentity()?.stackId;
const worker=coordinator.register(DELIVERY_QUEUES.mcpClientMirror, {
retryMs:5_000,pausePassiveMaintenance:true,onSettled:()=>{pending=true;},
async run(signal) {
pending=false;
const managedIdentity=getCloudRuntimeIdentity();
if(!managedIdentity)return;
if (!origin || !instanceId || !publish) return;
let enabled=false,publicationAttempted=false;
try {
enabled=true;
const assertCurrentAuthority=()=>{
const identity=getCloudRuntimeIdentity(),currentConfig=paperclipCloudConnectorConfigFromEnv();
if(!identity||identity.stackId!==managedIdentity.stackId||boundStackId!==undefined&&identity.stackId!==boundStackId
||identity.canonicalOrigin!==origin||config&&(!currentConfig||currentConfig.instanceId!==instanceId
||currentConfig.baseUrl!==config.baseUrl||currentConfig.environment!==config.environment
||currentConfig.signPrivateKey!==config.signPrivateKey||currentConfig.sealPrivateKey!==config.sealPrivateKey))throw new Error("MCP client mirror current authority changed");
boundStackId??=identity.stackId;
};
if(enabled)assertCurrentAuthority();
const publishEnabled=(request:McpClientMirrorRequest)=>{assertCurrentAuthority();publicationAttempted=true;return publish(request);};
const captured=await db.transaction(async tx=>{
await tx.execute(sql`SELECT pg_advisory_xact_lock(736721042)`);
if(enabled)await tx.insert(state).values({id:"public",revision:0}).onConflictDoNothing();
const [row]=await tx.select().from(state).where(eq(state.id,"public")).for("update");
if(!row)return;
if(row.enabled===enabled){
if(!enabled||!reserveFullRevision&&(!refreshPublished||row.receipt===null))return row;
await prepareMcpClientMutation(tx as unknown as Db);
return(await tx.select().from(state).where(eq(state.id,"public")))[0];
}
await prepareMcpClientMirrorPolicy(tx as unknown as Db,enabled);
return(await tx.select().from(state).where(eq(state.id,"public")))[0];
});
refreshPublished=false;
if(enabled&&captured&&fullRecovery)reserveFullRevision=false;
if(!captured&&!enabled)return;
if (!captured || !Number.isSafeInteger(captured.revision) || captured.revision < 0) throw new Error("Invalid MCP client mirror state");
const generation = mcpClientMirrorGeneration({instanceId,origin,revision:captured.revision,enabled});
const old = mcpClientMirrorReceiptSchema.safeParse(captured.receipt);
const baseRevision = !fullRecovery && old.success && old.data.instanceId===instanceId && old.data.origin===origin && old.data.revision<=captured.revision
&& old.data.generation===mcpClientMirrorGeneration(old.data) ? old.data.revision : null;
if (!enabled&&baseRevision===captured.revision&&old.success&&old.data.enabled===false) return;
if(!enabled){
if(!captured.remoteResponsibility)return;
const result=mcpClientMirrorResponseSchema.parse(await publish({version:1,instanceId,origin,revision:captured.revision,baseRevision:null,expectedRecords:0,operation:"disable"}));
if(!result.complete||result.receipt.enabled||result.receipt.instanceId!==instanceId||result.receipt.origin!==origin||result.receipt.revision!==captured.revision||result.receipt.generation!==generation)throw new Error("MCP client mirror disable was not acknowledged");
const [ack]=await db.update(state).set({receipt:result.receipt,remoteResponsibility:false}).where(and(eq(state.id,"public"),eq(state.revision,captured.revision))).returning({id:state.id});
if(!ack)pending=true;
return;
}
const deviceKey=deriveLocalEncryptedInstanceSecret(`mcp-device-status:v1:${instanceId}`),keyId=mcpDeviceStatusKeyId(deviceKey);
const deviceRows=await db.select({requestId:devices.id,deviceCodeHash:devices.deviceCodeHash,clientId:devices.clientId,resource:devices.resource,ingressOwner:devices.ingressOwner,status:devices.status,expiresAt:devices.expiresAt}).from(devices).where(eq(devices.resource,origin+"/mcp/paperclip")).orderBy(devices.id).limit(1001);
if(deviceRows.length>1000)throw new Error("MCP retained device set exceeds canonical1000 admission bound");
const entries=deviceRows.map(row=>mcpDeviceStatusEntrySchema.parse({requestId:row.requestId,clientId:row.clientId,resource:row.resource,ingressOwner:row.ingressOwner,status:row.status,expiresAt:row.expiresAt.getTime(),fingerprint:mcpDeviceStatusFingerprint(deviceKey,row.deviceCodeHash,row.clientId,row.resource)}));
const deviceStatus={keyId,generation:mcpDeviceStatusGeneration({instanceId,origin,revision:captured.revision,keyId},entries),count:entries.length};
if(baseRevision===captured.revision&&old.success){
if(JSON.stringify(old.data.deviceStatus)===JSON.stringify(deviceStatus)&&(old.data.authorizationOrigin??origin)===(input.authorizationOrigin??origin))return;
await db.transaction(async tx=>{
await tx.execute(sql`SELECT pg_advisory_xact_lock(736721042)`);
const [current]=await tx.select({revision:state.revision}).from(state).where(eq(state.id,"public")).for("update");
if(current?.revision===captured.revision)await prepareMcpClientMutation(tx as unknown as Db);
});
pending=true;return;
}
const [count] = baseRevision===null ? await db.select({total:sql<number>`count(*)::int`}).from(clients)
: await db.select({total:sql<number>`count(*)::int`}).from(changes).where(and(gt(changes.revision,baseRevision),lte(changes.revision,captured.revision)));
if (!count || !Number.isSafeInteger(count.total) || count.total<0) throw new Error("Invalid MCP client mirror count");
const identity = {version:1 as const,instanceId,origin,revision:captured.revision,baseRevision,expectedRecords:count.total,deviceStatus,...(input.authorizationOrigin?{authorizationOrigin:input.authorizationOrigin}:{})};
const [owned]=await db.update(state).set({remoteResponsibility:true}).where(and(eq(state.id,"public"),eq(state.revision,captured.revision))).returning({id:state.id});
if(!owned){pending=true;return;}
let result = mcpClientMirrorResponseSchema.parse(await publishEnabled({...identity,operation:"begin",deviceKey}));
if (!result.complete) {
const resumeSequence=result.nextSequence,resumeDeviceSequence=result.nextDeviceSequence??0;
let cursor:string|undefined, sequence=0, sent=0;
while (true) {
signal.throwIfAborted();
const rows: {clientId:string;change:McpClientMirrorChange}[] = baseRevision===null
? (await db.select({id:clients.id,name:clients.name,redirectUris:clients.redirectUris,grantTypes:clients.grantTypes}).from(clients)
.where(cursor ? sql`${clients.id} COLLATE "C" > ${cursor}` : undefined).orderBy(sql`${clients.id} COLLATE "C"`).limit(200))
.map(row=>({clientId:row.id,change:{operation:"upsert" as const,client:mcpMirroredClientSchema.parse(row)}}))
: (await db.select({change:changes,client:{id:clients.id,name:clients.name,redirectUris:clients.redirectUris,grantTypes:clients.grantTypes}}).from(changes)
.leftJoin(clients,eq(clients.id,changes.clientId)).where(and(gt(changes.revision,baseRevision),lte(changes.revision,captured.revision),cursor?sql`${changes.clientId} COLLATE "C" > ${cursor}`:undefined))
.orderBy(sql`${changes.clientId} COLLATE "C"`).limit(200)).map(({change:row,client})=>{
if(row.operation==="delete")return{clientId:row.clientId,change:{operation:"delete" as const,clientId:row.clientId}};
if(!client)throw new Error("MCP client changed during publication");
return{clientId:row.clientId,change:{operation:"upsert" as const,client:mcpMirroredClientSchema.parse(client)}};
});
if(!rows.length)break;
for(const page of mcpClientMirrorClientPages(identity,sequence,rows.map(row=>row.change))){
if(sequence>=resumeSequence)result=mcpClientMirrorResponseSchema.parse(await publishEnabled(page));
sequence++;sent+=page.changes.length;
}
cursor=rows.at(-1)!.clientId;
if(rows.length<200)break;
}
for(const page of mcpClientMirrorDevicePages(identity,entries)){
if(page.sequence>=resumeDeviceSequence)result=mcpClientMirrorResponseSchema.parse(await publishEnabled(page));
}
signal.throwIfAborted();
const [current]=await db.select({revision:state.revision}).from(state).where(eq(state.id,"public"));
if(current&&Number.isSafeInteger(current.revision)&&current.revision>captured.revision){pending=true;return;}
if(current?.revision!==captured.revision || sent!==count.total)throw new Error("MCP client mirror changed during publication");
result=mcpClientMirrorResponseSchema.parse(await publishEnabled({...identity,operation:"complete"}));
}
if(!result.complete||!result.receipt.enabled||JSON.stringify(result.receipt.deviceStatus)!==JSON.stringify(deviceStatus)||result.receipt.instanceId!==instanceId||result.receipt.origin!==origin||result.receipt.revision!==captured.revision||result.receipt.generation!==mcpClientMirrorGeneration(result.receipt))throw new Error("MCP client mirror was not acknowledged");
const receipt=result.receipt;
let retained=false;
await db.transaction(async tx=>{
await tx.execute(sql`SELECT pg_advisory_xact_lock(736721042)`);
const [current]=await tx.select().from(state).where(eq(state.id,"public")).for("update");
assertCurrentAuthority();
if(!current||!current.enabled){pending=true;return;}
if(!Number.isSafeInteger(current.revision)||current.revision<receipt.revision)throw new Error("MCP client mirror ACK revision regressed");
if((receipt.authorizationOrigin??origin)!==(input.authorizationOrigin??origin))throw new Error("MCP client mirror ACK issuer changed");
const prior=mcpClientMirrorReceiptSchema.safeParse(current.receipt);
if(current.receipt!==null&&(!prior.success||prior.data.generation!==mcpClientMirrorGeneration(prior.data)))throw new Error("MCP client mirror ACK base changed");
if(current.revision>receipt.revision&&(!prior.success||!prior.data.enabled||prior.data.instanceId!==instanceId||prior.data.origin!==origin)){
pending=true;return;
}
if(prior.success&&prior.data.revision>receipt.revision){pending=true;return;}
if(prior.success&&prior.data.revision===receipt.revision&&!sameMcpClientMirrorReceipt(prior.data,receipt))throw new Error("MCP client mirror ACK generation changed");
await tx.update(state).set({receipt}).where(eq(state.id,"public"));
retained=true;
if(current.revision!==receipt.revision){pending=true;return;}
await installMcpDeviceCutover(tx as unknown as Db,receipt);
await tx.delete(changes).where(lte(changes.revision,receipt.revision));
});
if(retained){fullRecovery=false;reserveFullRevision=false;}
} catch(error){
if(enabled&&publicationAttempted&&!fullRecovery){fullRecovery=true;reserveFullRevision=true;}
pending=true;throw error;
}
},
hasPending:async()=>false,nextRunAt:()=>pending?Date.now()+5_000:null,
});
return {worker,instanceId};
}