Linux is an amazing operating system. I have written about it several times over the years. All of our company’s servers run on CentOS Linux (about a thousand) and a good chunk of our desktops and work stations run Ubuntu linux. The speed and reliability of the main linux kernel (Think engine of a car) is unmatched in my estimation. What is even more impressive about the Linux kernel is its unbelievably rapid pace of development. Herein lies our problem.
The linux kernel is developed simultaneously for so many different work loads that it is impossible to ship a Linux Distribution that is tuned for your specific workload. If I could use the car analogy one more time – Linux could just as easily be an all electric 50 HP car, or a 1,000 HP Dragster, or a Semi Truck. It literally can and is used to power your phone to the fastest super computer in the world. With that type of flexibility how do you wrangle the most power out of the linux kernel for your specific needs?
Unfortunately, right now the answer is – With a GREAT DEAL of effort. I mentioned earlier that we use CentOS. This is a free rebranded version of Redhat Enterprise Linux which is a server class linux distribution. Meaning it is geared toward customers with a heavy server workload. The problem is the linux kernel they use in CentOS is SLOW, outdated, and certainly not tuned to our workload.
In a way this is a big benefit for us. I know of no other hosting company on the planet that spends more time and effort squeezing the most out of the linux kernel than Bluehost and Hostmonster. Whenever we see ANY bottleneck in the system whether it be CPU, I/O Block Device, Network Block Device, Memory, and so on we find out EXACTLY what is causing the problem. When I say we find the problem, I mean we go down to the actual code in the kernel and see exactly where the issue is. Sometimes that gives us the answer we need to the solve the problem and other times it is a bug in the kernel itself that we need to create a patch for.
Using these techniques we have been able to solve disk I/O issues and many other bottlenecks that have plagued EVERY linux distribution that I have ever tried. Linux is extremely capable, but it needs brilliant people that understand all aspects of the kernel to achieve that kind of speed and performance from a stock kernel.
I feel that given the same exact hardware specifications that we could generally squeeze at least twice the performance from MySQL, Apache, PHP, and underlying CPU and IO intensive tasks compared with any server loaded up with a standard install of CentOS or Redhat Enterprise Linux.
While I think that is great, and it gives us a huge advantage over our competitors I don’t really think it ought to be that way. Many of these tunable options simply need to be applied to a system to make it faster, while other methods require a historical approach before tuning becomes effectively.
Regardless, my opinion is that the linux kernel DESPERATELY needs some software that will real time evaluate you servers CPU, Memory. I/O bottlenecks and makes these adjustments for you on the fly based on a constantly changing workload. This should have been written a long time ago. Having a thousand different tunable options is great if you know what every one does and how it affects every other setting. My experience is that 1% of the users know about 10% of the possible settings to change and the rest never gets touched. Thats a shame, because as fast as Linux is, as long as these “default” kernel settings are in place you will NEVER get the performance that you could out of existing hardware without your own kernel guys on staff (Like we do).
Realtime kernel adjustment by the kernel itself for the workload being operated on is what the kernel needs now. I can tell you that we have some aspects of this done already. If nobody else steps up the plate, we may release our own version of this to the community at large to build up and use for the betterment of all.
Matt Heaton / Bluehost.com