A BIG speed and stability improvement!

We have worked so much lately on many internals to our hosting platform. Sometimes I worry that our customers think we are standing still when they don’t see a lot of outward/customer facing changes. However, we have been feverishly working on improving the overall stability and speed of our platform.

We have nailed down CPU overages, memory overages, and disk i/o bottlenecks (i/o was the hardest one btw). We have delved deep into the dark art of linux process scheduling and have come away mostly unscathed :)

All of this backend work has made a tremendous difference to our customers, but now we have something new to announce. MySQL process scheduling protection. I know what you are thinking… That doesn’t sound exciting at all, but to us computer nerds its a big deal!

Basically a single rouge php/perl/ruby script utilizing MySQL can consume all the resources on a server and never even show up as using much CPU at all. I have written test scripts that consume less than 1% of the total “cpu time” yet leave a machine with 16 cores with 0% idle time on all cpu/cores and thousands of backed up processes. I can replicate this on virtually every hosting company that I have test accounts on (And I have a LOT of test accounts on competitor’s servers). This isn’t a rare thing that happens. Its VERY common, and many popular plugins for WordPress and phpBB cause this to happen very frequently.

Surely something like this wouldn’t be allowed to happen in a mature multiuser environment such as Linux! When I first discovered this bug I didn’t think it was wide spread. After running MANY tests and spending a lot of near sleepless nights proving my theories I discovered that it was happening hundreds of times per hour for short periods of time across all our servers. This causes short delays (Usually 5-60 seconds) that are very very difficult to track down after the fact.

The issue really is with MySQL itself, but it also is because of a serious design flaw in the current linux process scheduler. I am intentionally being vague on the specifics of the problem because I feel the fix we have developed will give us a substantial competitive advantage. Without question we will at some point in the future release this code to the community, but for now we will continue to use it in house to the benefit of our loyal customer base.

This speed and stability fix will impact directly (In a negative way – meaning that coding problems will have to be fixed on the customers side before their sites become usable) about 400 customers out of 547,000 (At the time of this blog post). Indirectly, everyone else will benefit greatly because it literally makes the difference for many of our servers from being 0% idle (Totally overworked) and 50% idle.

It is not yet live on our production environment (Except in a few controlled cases), but will be completely live on our system by approximately Wednesday of next week (March 31st, 2010).

Don’t expect a huge change, but certainly expect and demand stability from us as your hosting provider. Just remember that when you don’t have slowdowns and problems it’s NOT by chance. We kill ourselves trying to make Bluehost/Hostmonster the best shared hosting on the planet. We think we have succeeded, but if you don’t think so then the job we are doing isn’t good enough. Either way we will continue to do our best to solve the biggest issues as we see them. Thanks again for your business and your encouragement.

Matt Heaton / President Bluehost.com

82 Responses to “A BIG speed and stability improvement!”

  1. Juan Carlos says:

    I am very thankful for the opportunities that your excelent service offers.

    There are no necessity of front-end changes to realize the big efforts of your engineers. ( i hope all you enjoy it to )

  2. Ron says:

    I have several sub domain names. Until approx. 2-3 weeks ago, I had my first down time experience which lasted for over 45+ minutes. I couldn’t recall a single down time with hostmonster servers in the last 3 years. However, after talking with technical support staff – they identified the concern and gave me an approx. time the sites would be back up. The tech. support staff is the best and the reason for my continued patronage (including PHP & MySQL support)!

    Thanks…ron

  3. whmitty says:

    My Hostmonster site has experienced zero downtime for about a month. Typically the longest period I could count on was 2 weeks. I’ve been with Bluehost and Hostmonster for 3+ years and have experienced nothing but timely responses to the few issues I’ve had. Keep up whatever you and the crew are doing as it seems to be improving an already excellent service. Thanks!

  4. Mike Goitein says:

    I came to Bluehost originally through WordPress, and I have to say that my experience has been consistently positive. Not only for myself, but for my friends and family members that have needed hosting, and whom I’ve referred to Bluehost have all been happy. I find everything set up in a way that enhances flexibility, productivity and ease of use. Keep doing the great job that you’ve been doing, and I’ll keep being a champion for your hosting!

  5. [...] And, they’re not sitting still – check out this blog post from Bluehost founder Matt Heaton about the latest improvements. [...]

  6. Rewa says:

    All of my websites are working fine and never had a warning. But as everyone is discussing about files limitation and CPU throttling in Host monster so I just checked how my account with 10+ wordpress blogs is doing and what I saw was a warning:-
    “During the past 24 hours your account has been throttled for a total of 10997.849 seconds.”

    When I clicked on Current Processes, I see:-

    “No current processes found”

    So, I contacted cust support ( Jack) for assistance and he told that its upto me to fix these problems myself and then he closed the chat without letting me explain!

    So, I removed 6-7 of my high traffic websites to Hostgator. And removed those websites including files and databases from hostmonster completely.

    Today , if I check the CPU throttling I see same message !
    “During the past 24 hours your account has been throttled for a total of 10997.849 seconds.”

    So, I certainly think that there are bugs in the throttling system you have developed. Consider reducing load on servers and optimize at your end.

    I am sure CPU throttling message will remain there even if I host only 1 domain with 5 visitors per day on this hostmonster account.

    My hosting account only has wordpress blogs with some basic plugins that are available through wordpress website..

  7. G. David Frye says:

    Matt,

    Another customer, “bidmyreno”, responded above with something that is proving to be a fatal flaw in our BlueHost service, serious enough that we are considering moving elsewhere:

    The limit on emails/hours is ridiculously low.

    Our domain is a unified service for PTAs, sports teams, and other parent-run activities in a district of 18 schools. Our primary uses of BlueHost are for activity web pages and mailing lists. A typical PTA mailing list has 400-600 subscribers. I use “mailman” to set up individual PTA lists because I have found that a) the people managing the lists do not have the time or expertise to learn a complex platform, and b) (more importantly) it allows us to subscribe people directly because any subscription process that requires an email authentication/web registration loop has been shown to lose 25% or more of our population.

    Mailman’s flaw is that it doesn’t throttle email. But even if it did, we can’t be in a situation where the messages that are sent out might not be received for several hours. Unfortunately the more common situation is that the mail is rejected outright due to email/hour limits exceeded. I was able to convince your support staff to increase that limit to 500/hour but I also elected – until we can make a decision about staying with the service – to limit the number and size of the lists I have created for our parent groups.

    I understand that you are concerned about a customer using the service as a spam platform. However, our need is legitimate. I can not any longer in good conscience restrict access to the groups that want to use our domain, and the limits are simply impractical. I also use Dreamhost and they have both mailman _and_ no practical limits to the amount of email we can send. When you are considering performance issues – which appears to be much of the time, based on your blog entries – please also think about this problem. Thanks.

  8. Rob Gundling says:

    All the more reason we stick so many of our customers on your servers. Thanks for the continues great work behind the scenes!

    Rob
    -=b2b solutions

  9. Jeff Gossett says:

    Matt,

    Here is something you wrote in August 2007

    Some companies just don’t get it and probably never will…

    Matt Heaton / President Bluehost.com

    Maybe your company is becoming like Google. Google was the company you referenced in that article. I just got the, “It’s our way or the highway” from your company. I wrote a letter to the feedback@bluehost.com address. Hopefully you will read it.

    I was told that upper management checks that email from “time to time” so they might not get it anytime soon.

    Also, I was told by James (your supposedly man in charge of all domains), through Mike, that he didn’t have time to talk with me and that he didn’t have the patience to talk with me.

    I asked that if I had patience why didn’t he, and Mike said, “Because he is human.” Last I checked I am human as well. When does the arrogance of man give them the right to be impatient with someone because we are human??

  10. mitch says:

    Glad to hear about the changes. I enjoy using bluehost for a number of reasons. Ease of use, compatibility, and features to name a few.

    Thanks!

  11. We’ve used Bluehost for a couple of years. Have only had a few outage problems. My one suggestion from reading the blog is Matt, you should respond back as people leave comments. Maybe you do, and just haven’t had a chance. This is the first time I’ve been here.

  12. [...] changes announced today have a great impact on the quality of service, something Matt Heaton seems pretty over the moon about. They’re not changes you’ll notice in your daily use, but this is the kind of [...]

  13. Excellent job Matt, I hope this helps with some of the DoS issues we’ve experienced in the past. All in all, I think Bluehost is probably the best hosting service around.

    I wrote your announcements into a blog post of mine this morning (Bluehost Adds New CPU & Memory Protection Features).

    Hope you don’t mind my quoting you!

    Excellent improvement to the service, I’ll continue recommending you.

  14. MDS Medical says:

    We host a number of domains on our shared server, with relatively few problems to date as compared to previous hosts! We migrated all of our medical practices over to Bluehost awhile back, and adding new ones daily. I have had some issues with simple and advanced DNS configuration as of late, but your support people are willing to make the changes manually so it’s not a HUGE deal, but I do like to have control and the convenience of updating myself. Ajax errors.

    Overall, the response time is great, the reliability is great, and I am happy!! Keep it up, and hope to hear from you more than once a year!!

  15. Kevin says:

    Hi, THanks for the new feature of bluehost ! I’m going to buy another hosting account with blue.

    Cheers.

    Kevin

  16. Great news.

    I must say that I’m quite happy with BlueHost but I hope the frequent downtimes for the past 3 months will end soon with all these upgrades..

  17. I very much appreciate this! I knew that I had done my part and had run “explain select” on all of my application’s queries to confirm that all necessary indexes were in place, but still I’d get the occasional call that the on-line system would freeze for a minute or so. I hope that MySQL/PHP performance will be more reliable. Thanks for the effort!

  18. Lyle says:

    Chris, ssh seems still to be working for me. I just tried it and got in within a few seconds.

  19. Mark Newsom says:

    Our site stopped working yesterday afternoon and the only way it is up and running now is because I had to do a redirect to the IP address. No one there (and I spoke with 4) had any clue what was going on and could only say it was somthing messed up in our code. So, we downloaded everything and ran it on a stack on a laptop and everything works fine. Also what doesn’t make sense is why would it be something in our code if it works with the ip and not the domain name? Sounds more like a datacenter issue and probably has something to do with DNS.

  20. maio says:

    Great service.. thanks for the great service I am having hosting my domain.

  21. matt says:

    I completely agree. I’m sorry for any problems you have had. Please feel free to email me directly at matt@bluehost.com. Please reference this comment along with any problems you are having and I will try my best to get them taken care of for you.

    Thanks,
    Matt Heaton

  22. joe says:

    Sounds like a huge bug and security issue with Linux/MySQL

  23. steve says:

    I am having issues with wordpress plugins/script errors in general (like as we speak) But service with bluhost is the best I have encountered so I am not going to lose sleep right now.

  24. Good to notice that you are enhancing your hosting features for good. I can also see that you are at #1 position on WordPress recommended web hosts.

  25. Claudine says:

    thanks for the great service, I am very thankful for the opportunities that your excelent service offers. Excellent job

    Thanks claudine

  26. Karl says:

    Hi Matt,

    I’m just looking at your blog for the first time and thought this was a great place to make my comment. I have used your service to host my websites for several (I think 9) years and over that time I have frequently referred friends to your service. I have gone as far as (I’m an IT guy) actually moving their blogs for them etc… It finally dawned on me that I never fail to plug you guys so I signed up for your affiliate program. Thanks for giving us that opportunity. I chose this post as you are talking about improvements to your rock solid service. Every customer I have brought you has been earned several times over, thanks for all the attention to us users. I have one account and pay a very nominal fee, but when I have a problem your guys are on it as though I’m the big fish. I will gladly continue to promote bluehost.com for years to come if your service levels remain or improve as they have continued to. Great job!

  27. RW Wood says:

    Great article. However, paragraph 4, “rouge” should be “rogue”. Rouge is something you put on your cheeks. :-)

  28. Brian says:

    I just found your blog and was interested in this post. Are you saying that deflate is already activated on our accounts now, or do we have to activate deflate? If so, is there a tutorial on how to deflate?

  29. greg schulz says:

    Hello Matt,

    First I have to say that I have been generaly pleased with your service and on going improvements over the years. However, as my sites on bluehost are currently down due to some type of outage at your facility, I figured a few minutes on your blog are appropriate to discuss single points of failure and resilient data centers.

    You see, I have two books whose sites are hosted via bluehost.

    One of those books is called “Resilient Storage Networks – Designing Flexible Scalable Data Infrastructures” (Elsevier) and the other is “The Green and Virtual Data Center” (CRC) where if my sites were not down to a facility disruption, could be found at http://www.storageio.com/books

    The reason I mention these is that in those books as well as content on my site, there is plenty of coverage of eliminating single points of failure (SPOF) from a server, storage, networking, hardware, software and best practices perspective.

    Send me a note with your mailing address and I will be happy to send you signed copies of both.

    Meanwhile, I hope your continued improvements help to reduce these types of service disruptions.

    Keep up the good work.

    A mostly satisfied customer!

    Cheers gs

    Greg Schulz
    Server and StoageIO
    http://www.storagio.com
    http://www.storageioblog.com
    twitter @storage

  30. [...] I took a different approach (you could have read it here however I just checked and it has been deleted). My comment on Matts blog post took a week or so to [...]

  31. [...] 12:08 am and is filed underneath Uncategorized. You can follow any responses to this entrance by a RSS 2.0 [...]

  32. Ligo Saint says:

    I certainly think that there are bugs in the throttling system you have developed. Consider reducing load on servers and optimize at your end.

Leave a Reply