Reliability, reliability, reliability…

Well, in the last week we had a major DDOS (Distributed denial of service attack). It hurt our network, and caused 2 outages. One for about 20 minutes and one for 45 minutes. That amount of downtime is absolutely unacceptable from a customers point of view. I totally agree!

I sent out a post to all our users letting them know what happened and reiterated our commitment to solving the problems that we are having. We received many positive comments saying how our customers appreciated us telling them what happened, and passing on their understanding for our situation. We also received a few emails stating that we had lost our focus and were growing at the expense of current customer satisfaction. While these emails were a very small percentage of the whole I take them very seriously. Let me address some of the issues we have and what we are going to do about it.

1 – DDOS attacks. Without going into too much detail we are giving this issue our FULL attention. While nothing can stop all DDOS attacks there are many things we can do to beef up our network beyond what we currently have. Don’t get me wrong, we have one of the most robust networks out there, but there is always more than can be done. We have already ordered some pretty cool new hardware that will help us get there. We are taking this seriously, and will do what we can to prevent further attacks.

2 – Individual customer overages – We run a shared environment. This means that you are on a server with many hundreds of other users. You have to be a good neighbor or everyone has a poor experience. We have been struggling with the balancing act of giving our users what they want and keeping their usage under control so that the environment we run is fast for everyone. In the future we will limit the cpu usage for the 1% that eat up the resources of the other 99%. We have been doing this all along, but have given more than we probably should to that 1%. This will make some customers angry, but for the vast majority it will mean improved uptime and greater speed for their sites.

3 – Proactive administration vs reactive administration – Sometimes we find ourselves in the reactive mode. Fix it when their is a problem. This isn’t right. We have set concrete plans in place that will make it so that each time a server goes down we have the information to know exactly what caused it and how to fix it so next time the same thing doesn’t happen. Many large hosts get caught up in the “reset the server” mode, and don’t really fix the problem for the long term. We will strive to fix it for the long term.

We want Bluehost to be not only the best hosting with regards to plans and service, but from a reliability standpoint as well. I would be interested to get your comments over the next month or so as we implement these new ideas.

Thanks,

Matt Heaton / President Bluehost.com

43 Responses to “Reliability, reliability, reliability…”

  1. Eric says:

    Thanks Matt.

    This blog is a great PR vehicle for BlueHost. I feel like I know you just from reading your posts, and understanding the focus of the company. I was already impressed with BlueHost, but I am even more impressed now.

    Kudos!

  2. Nascency says:

    Great post, Matt! It looks like you’re taking some very good steps, here.

    DDoS is a fact of life on the Internet. So many folks are quick to blame the host when there’s downtime. They don’t consider that any webhost has vulnerabilities. There will always be script kiddies and other idiots who will do their utmost to make all our lives miserable.

    I really like Bluehost and I’m glad every day that I found it. :-)

  3. Minh Nguyen says:

    I know this might be a bit much, but I know whenever ebay goes down, they have one day of insertion fees waived.

    What you guys should do is give away a month of hosting if servers ever go down more than X minutes. This should be supplemental to all the other things you mentioned in your post (since any host with a consistent record of downtime will lose customers – no matter how many months they give away).

    It’s just a thought. I still love BlueHost (assuming we don’t have anymore of these downtimes).

  4. Chris Fay says:

    Keep up the good work!

    It is nice to be kept in the loop when things happen. I have been with a few other hosts and none offer such high standards in both customer service and features which is crucial to me as a consumer. My site is rarely down and when it is, you are on top of it.

    Thanks a ton,

    Chris

  5. Matt. from our perspective at Victoria Johnson Consulting, you are doing many things right. The BlueHost network is by far vey reliable, and I’m glad you are putting strategy in place to stay in a proactive mode. We tried many before BlueHost, and as long as your servers are running (and that we know will be a long winning streak), we’ll be with you.

    Keep up the good work!

  6. Christine says:

    Hi, I’ve been a customer of yours for a little over 4 months now, and I must say, that i’ve been very satisfied with your services. Nothing on the internet can be up 100% of the time obviously, but the nice thing about you is that you inform us if something does go wrong. Which is more then I can say for the majority of hosts out there. You have excellent packages ( I’m blown away by how much bandwidth I have, and hope it does not change :P ), and your support team is wonderful, patient, kind, and respond quickly. Wouldn’t ask for much more. My last host, offered almost as much as you do, but would not allow more then 5 people to view the website at one time, which was never mentioned anywhere o ntheir site. I find you people to be very honest, and I appreciate that the most. Thank you so much for providing such a wonderful service. = )

  7. iM_iNet says:

    “Life can be very dishearthening someone is always going to try to bring you down.”

  8. Daniel says:

    Thanks for the update! This personal approach is really a great service, my former host could learn quite a lot from how you treat your customers.

  9. sjfeldman says:

    What are you doing about the 1% eating up the 99%? And how will you stop another DDOS attack? No answers are given.

  10. [...] Decir que navegué por Internet y no encontré algo nuevo e interesante equivale a decir que perdí mi tiempo. Hace un par de dias me topé con el blog de Matt Heaton, el presidente de Bluehost, la empresa que aloja esta página. Despues de agregarlo a bloglines y leer un par de posts sobre hosting y Bluehost llegué a la conclusión que este tipo sabe lo que esta haciendo y que lo hace muy bien. Bluehost es una de las empresas más grandes de hosting y con una confiabilidad y disponibilidad superior a la media, anteayer fueron atacados con DDOS y la notificación por correo llegó inmediatamente. El incidente no pasó de 20 a 30 minutos, sin embargo para una empresa que se jacta de tener un uptime arriba del promedio esto es inaceptable. Luego el incidente fue explicado en este post. [...]

  11. Matt,

    You & your team at Bluehost.com should be commended for your upfront attitude about customer service, the dialogue you use to relate your ideas and plans, and your overall dedication to reasonably-priced hosting for all sizes of companies along with superior customer service and support. Thank you! Your blog page about Bluehost.com is an amazing idea! Good luck & keep up the fantastic work!

  12. rickles says:

    Matt:
    First off I was shocked to see Bluehost down. Even your home page was down, but I was really impressed how fast you got it up and running. You have a great staff. I have needed assistance from time to time and your techs are first rate. I have been with Bluehost for several years now and I can say without hesitation, I would not consider any other Company.
    Two thumbs up!

    Rickles

  13. koallalays says:

    Great :)
    Thanks for letting us know whats going on with the server and its downtime!!! keep up good work :))))

  14. Ben Adlin says:

    Matt, about a week ago I posted on my blog the troubles that I had been having with my site’s downtime. I had been experiencing a few hours on and off throughout the day, and if you think 20-45 minutes is “unacceptable,” I think you’ll agree that many hours within the course of a week is borderline absurd.

    I actually just finished a post to commend you and BlueHost, as I’ve been impressed with both your public addresses (like this) and the speed and amiability of your customer service staff. However, when I went to publish it, I found that my site was once again down (and still is). I hate to call you out in so public of a forum, but this is getting kinda nuts. Every time I hear back from support, they say that they’ve fixed the problem, but every time it ends up going down again within the day.

    I believe that your company really does hold itself to high standards, but it’s really hard to keep talking BlueHost up when I’ve been experiencing so many problems with my service. Thoughts? Suggestions?

    Thanks so much!

    - Ben

  15. Hercules says:

    Hi Matt,

    I’m a little worried on the part of squeezing something out of the “big users”
    I run a website filled with partypictures and I’m pretty sure that 99,99% of the time it runs smoothly, but like 2 times a week when I update Coppermine has to resize the uploaded pictures and create thumbnails which seems to be hard working for a few minutes.
    When I have to update like 60 or more pics I get a Highload message and have to wait 2 minutes or so to finish the batch.
    I don’t hope that this uploading gets capped a lot more because then my website updates would become quite a pain in the $#%.

    Greetz

  16. lori says:

    Just wanted to comment that i have been a customer for just about a year now, and have no issues with your service at all. You are extremely reliable, and am glad that I made the decision to move my domain to you to host. DDoS are very irritating.. i would look for the reason why you were attacked though. It could happen again :/

  17. Matt says:

    I’ve been through my fair share of hosts, for fun and work. My first BlueHost site went up a few months ago, and I have to say from the post-order verification/welcome call to the regular detailed updates like this one, I have had a great customer experience. BlueHost will definately be seeing more business from me in the future. DDoS attacks can be a huge pain to straighten out, best of luck keeping the hackers off your back!

  18. Reading the blog, the first few paragraphs of this article, impressed me that you actually contacted all customers and told them about the downtime.

    My host will not even admit that their website was also down for quite some time. (integrahost)

    However, bluehost is also part of hostmonster and a few others, because the contact info is the same and the sites look similar, is this true?

  19. Cory says:

    You guys are awesome.
    WebDAV!!

  20. dear Matt,
    Kudos for the blog and thanks for outlining the steps you and bluehost are taking against future DDoS attacks.
    I have to say that I was very pleasantly surprised to discover your blog, it added to my very favorable impression of bluehost – past experiences with previous hosting companies made me hesitant to believe comments and guarantees of “great customer service” but you guys deliver. I can’t believe that it took me so long to discover your company!

  21. Steve says:

    Same here, i was shocked. BlueHost must be running VoIP because even the phones where down when i tried to call them! LOL

    with in 25 or so minutes it was back up. good work!

  22. James Mushen says:

    Matt, I have waited over a week for a dedicated IP- you guys ran out-
    which is hard to imagine—-I am told today that it will be another week to two weeks to aquire the hardware for the additional IP block—-DDoS and being down for a bit is a big deal—-and hard to anticipate—- What about being down for 3 weeks due to poor planning???

  23. Joao says:

    You rock.

  24. Minh Nguyen says:

    In addition to bluehost it seems you own:
    hostmonster.com

    How come you rebadged the company? Competitive advantage? I like the website design at hostmonster.com better.

    Seems like you’re also a part of pagewizard.com.

  25. Ivy Lee says:

    First, let me thank your at least admitiing that there is a problem and commiting yourselves to fix it. I do need to argue with the statement of a 45 minute down time, I have found my site and your services down repetatdly throughout the past two weeks. I appreciate that no host can be up 100%, but I hate to think of the lost customers during thse down times, net users don’t usually return to a site that was down the first time thney tried it. Do plan on at least a token compensation?
    That grumble aside, I have been very happy with the service, support and bandwidth I get from Bluehost, keep up the great work and knuckle down on the slip ups and I think you’ll be fast on your way to the number one hosting service out there!

  26. Jamie says:

    I got this email, it’s good to know that you guys took charge right away.

  27. Zac says:

    I just found your blog!

    this is a nice little addition.

    I’m a little confused about the 1% 99% issue…..

    As i develop more of my sites and start to garner more and more traffic on my accounts (which actually promote Bluehost through the affilliate program), you are saying that it will become more and more likely that people won’t be able to see my websites?

    That doesn’t seem right…

    What is the requirement to get into that 1% category?

    I find that there are rather frequent occurances where my site isn’t available (mostly during my peak traffic hours). While on my other account which gets zaro traffic is still going…its ussually only for a few minutes, but that all equates to lost rev.

    Anyways, i’m glad i found your blog.

    maybe we could get some actual categories on here that handle specific questions about aspects of your services? Perhaps have a few articles by your crack support staff or som’n.

    Perhaps even a featured site here and there showing what is being done with your network.

    I think that would be a great way to cross promote the customer base blogs and client knowledge base as well as create a GREAT resource for us. (i know i would come here first before running to google, sitepoint, sourceforge etc…)

    Of course, though, is uppose that might increase traffic to much and then we would experience slowdown…..*sarcasm*

    overall, very pleased with Bluehost…

    p.s. glad you got those misspellings of the company name, i’m always hitting the wrong keys!

  28. chrys says:

    Thanks for being there as our “BLUE HOST!” as usual.

  29. Chris says:

    I can honestly say, that I do not recal receiving a message about the downtime. I’m also confident, that my downtime was more than 20 – 45 minutes.

    Either way, I’m still happy with bluehost, and goodluck with preventing these events in the future.

  30. NICE JOB MATT and the whole Bluehost team! Bluehost is still the best hosting deal on the Internet. Count me as yet another satisfied customer.

  31. Phil Vouers says:

    One thing that troubles ME now with your security is the inclusion of the yahoo banner into my cpanel. We should have NO offlinks in our cpanel don’t you think??

  32. Donald Coors says:

    I run a site for a young and upcoming musician. He has had many Music Executives view his site including some of the top producers from Paramount Pictures. All of them have complimented how well the site works. Many have said it works better than many of the big artists site. Obviously the design of the site had something to do with this, but to me a main benefit is having BlueHost as the host. The reliability has been fantastic and I used tools off your help menu system to to the writing for the streaming video and audio files.

    Keep up the good work!

    Thanks!!

  33. Doug N says:

    20 and 45 minutes and you think that is unacceptable? WOW! I am glad I moved to bluehost.com. I was formally with hostreflex and I once had an outage of almost 4 days. Not one of my tech support requests/emails were ever answered. Hence the reason I switched hosting providers. So far I like what I see/hear/read.

  34. [...] Well I now have all most major hosting a bluehosting.com. I can’t say I miss hostreflex.com at all at this point. Granted it has only been about a week and who knows if the future holds a 3-4 day outage like I would sometimes get with hostreflex (and no replies to my tech support emails BTW). I then came across this blog posting from the bluehosting owner. [...]

  35. Keep the uprgrades and improvements coming. My website performance seems better than in the past, it used to be a little slow to load. All I ask is that you keep my website up and loading fast.

  36. Keep the uprgrades and improvements coming. My website performance seems better than in the past, it used to be a little slow to load. All I ask is that you keep my website up and loading fast.

  37. Rob Jones says:

    Glad to see you’re improving the service. Hosts really should do all they can to maintain uptime, it’s frustrating to have sites down when you rely on them for income. Overall I like your hosting a lot, unlike dreamhost whose reliability is a joke lately.

  38. Thanks for having this blog.

    I need more information about that 1% who are causing trouble for their shared hosting co-residents. My ecology organization wants to start serving eco-news via video, so we expect to use every bit of our 400GB downoad bandwidth. (That is about 2000 15-min news stories delivered.) Does that make me into one of the 1% of trouble causers? I think all your customers need a clear list of what “causes trouble.” Given specific info, we could be better neighbors.

  39. Sierra says:

    45 minutes? You think 45 minutes downtime is bad? Ipower’s been down now for over SIX HOURS!!! And it’s not even the first time this month it’s been down for a multi-hour stretch. Obviously, I found this blog while researching new servers :) However, I was also threatened with a cancellation fee if I terminated my agreement early, so I’m simply letting my hosting agreement run out (September), and then never ever ever going back.

    The one thing Ipower offers that I haven’t seen at Bluehost and I think is a fabulous add-on is the domain privacy registration — the WHOIS thingy. Ipower offers it free, but if it were available at Bluehost for a nominal fee (coupla bucks a year sort of nominal) I for one would definitely make use of it.

  40. Dan says:

    Matt, I’ve been a customer of Bluehost for about 3 years now and absolutly love it. Have you guys ever thought about using something like a Prolexic for DDOS protection?

  41. Wang Cheng says:

    I agree with your words.
    I use your hosting for 3 months, which is excellent.

  42. [...] the same post Abssorb quotes one of Matt Heaton’s blogs Reliability, reliability, relibility… 3 – Proactive administration vs reactive administration – Sometimes we find ourselves in the [...]

  43. Kim says:

    Reliability is so key these days when talking about web hosting. It is the number one quality I look for when choosing a host. This is the most vital attribute a host can have along with great server uptime.

Leave a Reply