zymr-dissecting-docker-swarm-elastic

Docker and Elasticsearch both have the properties and capabilities to function hand-in-hand in an optimal fashion – due to their elasticity. As most developers are aware already, running Elasticsearch clusters on Docker Swarm is relatively difficult due to the fact that the user must specify their IP addresses of ALL master nodes in order to join the cluster. Not only is this is a hassle, but it is also relatively time-consuming. The reason behind this is because Elasticsearch removed their multicast discovery option and incorporating it into a plugin – users must have this plugin in order to benefit from the features. Furthermore, the IP addresses that are required are not visible, for a lack of better words, to the user when one starts Elasticsearch. And Docker service rejects any use of a bridge or host network, leaving the user essentially caught between a rock and a hard place.

With a few workarounds, a user can bypass the difficulties of searching for other nodes through several specification commands. For instance, the developer can specify an Elasticsearch container to ask the node to search for other Elasticsearch nodes. Through this method, the IP addresses returned will be that of a random Elasticsearch node, therefore forming the cluster with little to no work.

While this method is a useful tactic, it does not fully solve the networking problem that many users face. Since Docker does not permit outside connections into overlay networks, due to an initial design limitation, the user will run into a port connection issue. The use of a proxy service to connect an outside network to the Elasticsearch network is a work-around of the design – or bug as many people like to believe. Users will need to utilize a proxy of their choice to discover the target service (through a series of tags). Note that these tags must be set as environment variables in regards to the Docker Swarm service for optimal effectiveness. If the user is not running a DNS configuration in the setup, a public DNS can be used for the virtual hostname. By incorporating these tactics, the user can allow an “outside” network to virtually connect to the Elasticsearch network without having to go to great lengths in order establish a simple connection.

Monitoring an Elasticsearch cluster through an ES image (along with specific plugins installed) will allow the user to check the health of the cluster frequently, or as needed. The idea behind this is that through the use of the resourceful options in Docker as well as a bridge network driver, one can essentially create a test environment for the user to determine what works effectively as opposed to other methods. For example, if an individual were to simulate a multi-node environment on a single server, the user can actually utilize Docker on a single server. Although this presents a formidable challenge to the developer, it is also a blessing in disguise as he or she can find success through a trial-and-error method.

This article was authored by Raseel Bhagat, who is a Software Architect at Zymr.

Recent Posts

Popular Posts

0 comments

Leave a Reply

© 2017, Zymr, Inc. All Rights Reserved.