Change DNS of a Sharded MongoDB Cluster

Intro

A MongoDB cluster consists of three components: shards in replica sets, mongos, and config servers in a replica set. All three components receive DNS information as parameters on startup via CLI or config files. Apart from that, the shards and the config servers store DNS information about themselves and about the other members of the cluster in local, admin, and config MongoDB databases.

In order to change the DNS names of the cluster, you need to modify both config files and the internal MongoDB databases. In this post, I explain how to do it step by step.

Change config files

The pattern of the new DNS names that I’ll use for this guide:

• mongosX.cluster.net - mongos;
• cfgX.cluster.net - config servers;
• shrdXx.cluster.net - shards;

All cluster components get their DNS names from config files or CLI. This is the first thing to change.

In the config file change the DNS name next to bindIp (mongos example):

net:
  port: 27017
  bindIp: mongos1.cluster.net

Make a similar change to all shards, mongos, and config servers.

The mongos also get information about the config servers from the config files. Change it too:

sharding:
  configDB: configReplSet/cfg1.cluster.net:27019,cfg2.cluster.net:27019,cfg3.cluster.net:27019

Here configReplSet is the name of the config servers’ replica set, yours might be different.

This is it for the DNS changes in the config files.

Change internal MongoDB data

This will be less trivial, but no rocket science either 🚀

Prepare cluster

Before we can modify the internal MongoDB databases we need to temporarily disable any security, sharding, and replication settings:

security:
  keyFile: /etc/mongoKey.pem
  
replication:
  replSetName: configReplSet

sharding:
  clusterRole: configsvr

If present, remove or comment the above lines in the config files of all config servers and shards.

Now restart the mongod services for the changes to take effect (Ubuntu example):

sudo systemctl restart mongod

Config servers

Connect to one of the config servers using the mongo shell.

Check the replica set configuration. From the mongo shell run:

use local
db.system.replset.find()

Now change the DNS names in the config servers’ replica set:

use local
cfg = db.system.replset.findOne( { "_id": "configReplSet" } )
cfg.members[0].host = "cfg1.cluster.net:27019"
cfg.members[1].host = "cfg2.cluster.net:27019"
cfg.members[2].host = "cfg3.cluster.net:27019"
db.system.replset.updateOne( { "_id": "configReplSet" }, { $set: cfg } )

Next, view your shards:

use config
db.shards.find()

Now change the DNS names of the shards:

use config

shrd1 = db.shards.findOne( { "_id": "rs1" } )
shrd1.host = "rs1/shrd1a.cluster.net:27018,shrd1b.cluster.net:27018,shrd1c.cluster.net:27018"
db.shards.updateOne({ "_id" : "rs1" } , { $set: shrd1 } )

shrd2 = db.shards.findOne( { "_id": "rs2" } )
shrd2.host = "rs2/shrd2a.cluster.net:27018,shrd2b.cluster.net:27018,shrd2c.cluster.net:27018"
db.shards.updateOne({ "_id" : "rs2" } , { $set: shrd2 } )

shrd3 = db.shards.findOne( { "_id": "rs3" } )
shrd3.host = "rs3/shrd3a.cluster.net:27018,shrd3b.cluster.net:27018,shrd3c.cluster.net:27018"
db.shards.updateOne({ "_id" : "rs3" } , { $set: shrd3 } )

Here rs1…rs3 are the names of the shards’ replica sets, your replica sets may have different names. In this example, there are 3 shards: 3 replica sets with 3 shard replicas in each set.

Do all of the above for the remaining config servers. The commands must be identical for each config server, so just reuse the commands from the first iteration.

Shards

The first part for shards is similar to the one of the config servers.

Connect to the mongo shell of any shard in any replica set.

Check the replica set configuration:

use local
db.system.replset.find()

Change the DNS names in the shard’s replica set:

use local
shrd = db.system.replset.findOne( { "_id": "rs1" } )
shrd.members[0].host = "shrd1a.cluster.net:27018"
shrd.members[1].host = "shrd1b.cluster.net:27018"
shrd.members[2].host = "shrd1c.cluster.net:27018"
db.system.replset.updateOne( { "_id": "rs1" }, { $set: shrd } )

The rs1 here is the shard’s replica set name. If yours is different, make sure to change it.

Next, we need to update the config servers’ DNS that the shard stores:

use admin
shrd = db.system.version.findOne( {"_id" : "shardIdentity" })
shrd.configsvrConnectionString = "configReplSet/cfg1.cluster.net:27019,cfg2.cluster.net:27019,cfg3.cluster.net:27019"
db.system.version.updateOne( { "_id": "shardIdentity" }, { $set: shrd } )

The shardIdentity value is a constant, you don’t need to change that.

Repeat all of the above to the rest of the shard replicas. Make sure to change shard replica set name (rs1) when you proceed to another replica set.

Wrap up

That’s all the changes we needed. Put back or uncomment the security, replication, and sharding sections in the config files and restart all the cluster components. Your cluster should now work with the new DNS names that you have assigned.

Final thoughts

We’ve looked at how to change the DNS names of all components of a MongoDB cluster. To do it properly, we had to modify both the configuration files and the internal MongoDB data.

To minimize the downtime, you can use configuration managers like Ansible and/or shell scripts. It might be possible to make the change gradually without downtime, but that would require extra effort, so if you can tolerate a short downtime, this approach is the way to go. Using this method I was able to change the DNS on my MongoDB cluster that consisted of 4 mongos, 4 config servers, and 6x3 shards in under 5 minutes.