The Linux Kernel = The Solution = The Problem

Linux is an amazing operating system. I have written about it several times over the years. All of our company’s servers run on CentOS Linux (about a thousand) and a good chunk of our desktops and work stations run Ubuntu linux. The speed and reliability of the main linux kernel (Think engine of a car) is unmatched in my estimation. What is even more impressive about the Linux kernel is its unbelievably rapid pace of development. Herein lies our problem.

The linux kernel is developed simultaneously for so many different work loads that it is impossible to ship a Linux Distribution that is tuned for your specific workload. If I could use the car analogy one more time – Linux could just as easily be an all electric 50 HP car, or a 1,000 HP Dragster, or a Semi Truck. It literally can and is used to power your phone to the fastest super computer in the world. With that type of flexibility how do you wrangle the most power out of the linux kernel for your specific needs?

Unfortunately, right now the answer is – With a GREAT DEAL of effort. I mentioned earlier that we use CentOS. This is a free rebranded version of Redhat Enterprise Linux which is a server class linux distribution. Meaning it is geared toward customers with a heavy server workload. The problem is the linux kernel they use in CentOS is SLOW, outdated, and certainly not tuned to our workload.

In a way this is a big benefit for us. I know of no other hosting company on the planet that spends more time and effort squeezing the most out of the linux kernel than Bluehost and Hostmonster. Whenever we see ANY bottleneck in the system whether it be CPU, I/O Block Device, Network Block Device, Memory, and so on we find out EXACTLY what is causing the problem. When I say we find the problem, I mean we go down to the actual code in the kernel and see exactly where the issue is. Sometimes that gives us the answer we need to the solve the problem and other times it is a bug in the kernel itself that we need to create a patch for.

Using these techniques we have been able to solve disk I/O issues and many other bottlenecks that have plagued EVERY linux distribution that I have ever tried. Linux is extremely capable, but it needs brilliant people that understand all aspects of the kernel to achieve that kind of speed and performance from a stock kernel.

I feel that given the same exact hardware specifications that we could generally squeeze at least twice the performance from MySQL, Apache, PHP, and underlying CPU and IO intensive tasks compared with any server loaded up with a standard install of CentOS or Redhat Enterprise Linux.

While I think that is great, and it gives us a huge advantage over our competitors I don’t really think it ought to be that way. Many of these tunable options simply need to be applied to a system to make it faster, while other methods require a historical approach before tuning becomes effectively.

Regardless, my opinion is that the linux kernel DESPERATELY needs some software that will real time evaluate you servers CPU, Memory. I/O bottlenecks and makes these adjustments for you on the fly based on a constantly changing workload. This should have been written a long time ago. Having a thousand different tunable options is great if you know what every one does and how it affects every other setting. My experience is that 1% of the users know about 10% of the possible settings to change and the rest never gets touched. Thats a shame, because as fast as Linux is, as long as these “default” kernel settings are in place you will NEVER get the performance that you could out of existing hardware without your own kernel guys on staff (Like we do).

Realtime kernel adjustment by the kernel itself for the workload being operated on is what the kernel needs now. I can tell you that we have some aspects of this done already. If nobody else steps up the plate, we may release our own version of this to the community at large to build up and use for the betterment of all.

Thanks,
Matt Heaton / Bluehost.com

12 Responses to “The Linux Kernel = The Solution = The Problem”

  1. The Linux Kernel = The Solution = The Problem…

    Linux is an amazing operating system. I have written about it several times over the years. All of our company’s servers run on CentOS Linux (about a thousand) and a good chunk of our desktops and work stations run Ubuntu linux. The speed and …

  2. Matt says:

    I always love your insight into the technical side of your business. As a sysadmin with no time to delve deep into the kernel I am both envious and excited by the possibilities. I think that by releasing your findings you will inspire others to take those innovations and expand them not only helping themselves but giving back to the community. I know this is a difficult decision as it is not just about community, but also about business.

    Once again thanks!

    Matt
    http://photos.millardfam.com
    http://twitter.com/gocyclones

  3. Insight on the linux kernel…

    Image via Wikipedia

    Found this great post in my RSS feads by the CEO of Bluehost.com and Hostmonster.com, Matthew Eaton. I love his blog and even if I wasn’t a customer of Bluehost I would still be following him for these kind of ge…

  4. Henri says:

    Sounds great! I’m sure the CentOS folks and linux devs would be very interested to see what you have done.

  5. Jason says:

    Matt,

    You are right, and I can only echo what others have said.

    I’ve said it many times. I love Open Source, including Linux, but the development relationships that bring about OS tend to be by their nature either symbiotic or parasitic. Developers are getting paid by someone; they have to eat. I always like it when a company that pays the developers really believes that the relationship is actually a symbiosis. You see yourself as part of the solution, and you don’t see OS development as a host-parasite relationship.

    I do hope you will consider contributing your code back to the OS community through contributing it back to the kernel.

  6. I’ve been messing around with Unix since 1977 and with Linux from the get-go. Unfortunately, I haven’t been forced to spend as much time with it as I need to stay fluent, but I have a lot of respect and admiration for it.

    I’ve put in my time with gcc and other things of the GNU ilk and have been programming in Php and mySQL for a number of years.

    It’s a great pleasure to have found your hosting company and moved my websites and databases over to your world. I appreciate your attitude, your utilities, and your great service personnel. Thanks.

  7. V. Bojkov says:

    Hello Matt,
    Can you publish or give us an idea about real benchmarks between CentOS4.7 stock kernel 2.6.9-78.0.5.ELsmp and custom compiled one BH is using from kernel.org (2.6.26-5_2.BHsmp) ? This will be really interesting to have an idea about your achieved improvements.

    Thanks,
    V.Bojkov

  8. Brent2 says:

    That’s an interesting thought. Bluehost kernal distro for Redhat. If it works, it might generate business. Plus, coolness.

  9. Brent2 says:

    Was thinking.

    MonsterKernal
    BlueKernal
    HostKernal

    I like BlueKernal best. MonsterKernal sounds like it’s big and slow while HostKernal is just, uh, a little stupid.

  10. Jose Bravo says:

    Is good to see people that really want the customer profits, dosn’t matter how hardest it’s (because kernel patching hard and take much time, monitoring, coding, patching, recompiling, testing, monitoring).

    José David Bravo Álvarez
    http://www.colombiahosting.com.co

Leave a Reply