mattheaton.com

The last several months we have been feverishly testing many different hardware server configurations trying to squeeze every last bit of performance from each of them. One of our main focuses have been towards improving MySQL performance and segregating MySQL users resources from one another so that a user that is abusing the system wont’ affect others on the same server.

The software side of this equation and related custom MySQL patches that make it happen will follow in a few days on this blog along with GPLed source code to help out. However, for this entry I want to focus on the hardware side of things.

How have we increased performance on the hardware side? We are putting in SSDs of course! SSDs or solid state drives have been getting a lot of attention lately – and rightly so! They are EXTREMELY fast. The problem with SSDs are that many aren’t compatible with certain raid controllers, many are slow, and many have extreme write delays. The ones we use have none of these problems. Even random writes which is typically the slowest stat for SSDs outperform any regular platter based drive we have ever tested.

To illustrated how fast our software and hardware combination is compared to just a few months ago I took a server with 500 clients on it and benchmarked it in every way I could. We then took five of these servers and combined ALL the users onto a single new server with our new platform with the SSD and MySQL caching system. With 2,500 users on a single system you would think MySQL would fall flat on its face. Instead it is anywhere from 5x to 100x faster in our multiuser environment.

MySQL used to be the number one process (and corresponding threads) with blocked I/O on our system in a 24 hour period. This had always been the case. On one of our new systems it doesn’t even show up in the top ten. It really is that much faster!

The only downside is that very few of our server setups have our hardware/software MySQL combination rolled out. We are ready to begin rolling this out to most of our servers over the next 2-3 weeks. For those users on servers that can’t be upgraded to this new system you will be migrated to new systems that can take advantage of this setup.

With a MAJOR bandwidth upgrade happening next week, the rollout of our new MySQL solution, and MANY custom kernel patches things are about to get a lot faster and a lot more stable!

I know for many of our users that were originally migrated from our old system to our new system that it was a painful process riddled with downtime. I am so sorry for those issues. The testing process we used was sorely lacking – the responsibility was mine and I fell short for those users. The migration process should be MUCH smoother going forward. If it is any consolation, all of the users that were migrated to our new system will be the first to get the benefit of the new super fast database implementation.

Thanks for supporting Bluehost and Hostmonster!

Matt Heaton / President Bluehost.com – Hostmonster.com

Linux is an amazing operating system. I have written about it several times over the years. All of our company’s servers run on CentOS Linux (about a thousand) and a good chunk of our desktops and work stations run Ubuntu linux. The speed and reliability of the main linux kernel (Think engine of a car) is unmatched in my estimation. What is even more impressive about the Linux kernel is its unbelievably rapid pace of development. Herein lies our problem.

The linux kernel is developed simultaneously for so many different work loads that it is impossible to ship a Linux Distribution that is tuned for your specific workload. If I could use the car analogy one more time – Linux could just as easily be an all electric 50 HP car, or a 1,000 HP Dragster, or a Semi Truck. It literally can and is used to power your phone to the fastest super computer in the world. With that type of flexibility how do you wrangle the most power out of the linux kernel for your specific needs?

Unfortunately, right now the answer is – With a GREAT DEAL of effort. I mentioned earlier that we use CentOS. This is a free rebranded version of Redhat Enterprise Linux which is a server class linux distribution. Meaning it is geared toward customers with a heavy server workload. The problem is the linux kernel they use in CentOS is SLOW, outdated, and certainly not tuned to our workload.

In a way this is a big benefit for us. I know of no other hosting company on the planet that spends more time and effort squeezing the most out of the linux kernel than Bluehost and Hostmonster. Whenever we see ANY bottleneck in the system whether it be CPU, I/O Block Device, Network Block Device, Memory, and so on we find out EXACTLY what is causing the problem. When I say we find the problem, I mean we go down to the actual code in the kernel and see exactly where the issue is. Sometimes that gives us the answer we need to the solve the problem and other times it is a bug in the kernel itself that we need to create a patch for.

Using these techniques we have been able to solve disk I/O issues and many other bottlenecks that have plagued EVERY linux distribution that I have ever tried. Linux is extremely capable, but it needs brilliant people that understand all aspects of the kernel to achieve that kind of speed and performance from a stock kernel.

I feel that given the same exact hardware specifications that we could generally squeeze at least twice the performance from MySQL, Apache, PHP, and underlying CPU and IO intensive tasks compared with any server loaded up with a standard install of CentOS or Redhat Enterprise Linux.

While I think that is great, and it gives us a huge advantage over our competitors I don’t really think it ought to be that way. Many of these tunable options simply need to be applied to a system to make it faster, while other methods require a historical approach before tuning becomes effectively.

Regardless, my opinion is that the linux kernel DESPERATELY needs some software that will real time evaluate you servers CPU, Memory. I/O bottlenecks and makes these adjustments for you on the fly based on a constantly changing workload. This should have been written a long time ago. Having a thousand different tunable options is great if you know what every one does and how it affects every other setting. My experience is that 1% of the users know about 10% of the possible settings to change and the rest never gets touched. Thats a shame, because as fast as Linux is, as long as these “default” kernel settings are in place you will NEVER get the performance that you could out of existing hardware without your own kernel guys on staff (Like we do).

Realtime kernel adjustment by the kernel itself for the workload being operated on is what the kernel needs now. I can tell you that we have some aspects of this done already. If nobody else steps up the plate, we may release our own version of this to the community at large to build up and use for the betterment of all.

Thanks,
Matt Heaton / Bluehost.com

True story – In 1945 there was a chicken in Colorado that had its head cut off (Sorry, gruesome I know). What’s incredible about this particular chicken is that the axe missed its jugular vein but left enough of the brain stem attached for this chicken to live. Not only did this chicken live, but it actually seemed to thrive. In fact, in the following two years the chicken actually put on 6 lbs.

This chicken was later named Mike and actually toured the country being shown at different shows and events. The chicken made its owner up to $4,500 a month (63 years ago!!!). People by the thousands paid 25 cents to see Mike and were amazed at how Mike seemed to live a completely “normal” life.

Mike was fed through an eyedropper, but other than that he simply went about his life. Even headless, Mike would happily “peck” for food blissfully unaware that he had no head.

Now this story is amazing to me for several reasons, but the one that stood out to me most was that a chicken with literally his head taken off could function as he did before. He was a drone. He did the same activities day in and day out but never had to “think” for himself.

Sometimes as I watch people that do the same thing day after day with no ambition or action to change their circumstances or their lives that they are no better off than Mike. We all have the capability to live a full life despite our circumstances, but it requires work, dedication, and proper planning. Instead, I believe so many are content like Mike to “peck” around and do their daily activities only worrying about what they are doing the next day.

We have a head for a reason – Lets learn to use it for our benefit. Don’t be like Mike that only lived day to day and was content. If you are in a situation that you don’t want to be in then use your head and improve your status. Quit running around like a chicken with your head cut off and do something worthwhile!

More Info On Mike

Matt Heaton / Bluehost.com