As web applications grow ever demanding with regard to overall resources required, monolithic server architecture has proven to be an expensive way to continually deliver more power through vertical scaling.
The problem being that a web application will often not need to use the full resources of the server hardware, but those resources still need to be available to take up the slack when needed. What this means is, that for much of the time, those big expensive servers are not being utilized to the fullest.
This is especially true when the overall application technology stack incorporates several different technologies such as Hadoop, Storm, Spark, SOLR, etc. Traditionally, each would run on its own server hardware, further exacerbating the “idle giant” problem.
What is needed, is some backbone technology that can facilitate the simple distribution of multiple dissimilar technologies as a single set of distributed application services across multiple large servers. Apache Mesos attempts to do just this.
Apache Mesos is a technology platform that attempts to solve the scalability problem, by taking technologies such as those mentioned above and turning them into frameworks that can run inside Linux containers. These containers can be hosted across a cluster server environment, virtualized as containers on a single piece of hardware, or any conceivable combination of both. This allows server resources to be leveraged more intelligently.
Apache Mesos works with many modern technologies such as Hadoop, Memcached, Ruby on Rails, Storm, JBoss Data Grid, MPI, Spark, and Node.js, as well as various web servers, database servers and application servers.
Mesos frameworks run on all machines in the cluster, with a single machine being designated as the master. All other machines are seen as slaves. Each slave has specific resource levels assigned to the framework. In theory, the resource limit for a Mesos framework is equal to the total of the slave machine resources. But because this is a distributed cluster, Mesos will only apply the resources needed to perform a specific task.
This means that for trivial tasks, many slaves may remain idle for the entire process. If the physical hardware hosting these slave framework is also hosting other frameworks, the resources remain free to be used. Meaning Mesos is intelligently extracting resources in the most efficient manner, and balancing resource usage across all frameworks that make up the entire cluster.
Hubspot utilizes hundreds of AWS servers to deliver its full product offering. By deploying Apache Mesos to its server infrastructure, Hubspot has been able to begin to significantly reduce the number of small servers it uses and replace them with far fewer large servers.
Traditionally, those large servers would sit with a lot of idle resources for much of the time. But Mesos ensures these spare resources are leveraged whenever they become available.
Hubspot has stated that there have been many benefits gained from deploying Mesos, and these include:
[See Also: NXlog, ElasticSearch, and Kibana]
Apache Mesos is actually proving that the old style of server farm, full of large, powerful machines can actually be more cost effective than server farms that use many smaller servers. A complete 360-degree turnaround on recent industry trends.
Mesos ensures that those large, expensive servers are used to the fullest. By running each framework across all of them, ready to extract processing power when it is available.
Everything you need to know about outsourcing technology development
Access a special Introduction Package with everything you want to know about outsourcing your technology development. How should you evaluate a partner? What components of your solution that are suitable to be handed off to a partner? These answers and more below.