Not so great the last couple of months….

When things are good they are great and when they are bad they are REALLY bad. The last 2 months we have been REALLY bad. I know many potential clients read this blog, but I have to be honest with our existing clients that we have been less than great the last 2 months and the problems lie squarely on my shoulders.

Today we had downtime off and on from around 10am-3pm. It was our garbage heap we call a router that did it to us again. We have had a list of things go wrong in trying to get our new router up and going. We have the new router in our datacenter but the fan tray is not the highspeed fan tray we needed and so it is holding us back. We won’t be able to put our new router into production until Friday night at around midnight. It was supposed to be up and running this Monday but evidently next day air means 4 days later to some people.

Here is the short list of majors problems that we have had to deal with in the last 2 months and what we are doing about these problems.

Problem – Major linux kernels problems with Redhat Enterprise 4. This affected about 40 servers. Solution – Finally built our own custom kernel that solved our multitude of issues.

Problem – Two seperate power outages that affected about half of our users. Solution – So far have moved about 80% of our users to a facility that has enough UPS and generators to alleviate the problem. We are signing a lease next week on a 5000 ft data center that will allow us to grow as well as provide colocation services for many businesses. The whole move will take around 45 days to complete.

Problem – Multiple outages affecting all our users. Slow connection speeds and dropped packets. – Solution – A new cisco 6509 router with a sup720-3bxl card to solve our routing issues. This is 95% of the problems that Bluehost has been experiencing. We have been trying to solve the router problem for a couple of months. I think we finally have a handle on this hairy problem!

Problem – APF firewall system running on all users boxes. This firewall software has been randomly blocking data ports for users across all our systems. There was no rhyme or reason to why this was happening and thus very difficult to resolve. Solution – Finally gave up on APF. It is about the worst software I have ever used. We have installed a new software solution on a dozen servers and have been testing it. It seems to have completely fixed our issues. It is being rolled out to all servers on Monday.
All of these issues have made Bluehost’s reliability in the last 2 months completely unacceptable. As you can see from above we know of the issues and have been working nonstop the last several weeks to get things back to how they should be. Thanks for those that have been patient. For those that haven’t been so patient I don’t blame you one bit! We will have things back to normal in a few days.

Thanks,

Matt Heaton / President Bluehost.com

96 Responses to “Not so great the last couple of months….”

  1. greg says:

    Kudos on your approach – I think current and potential customers appreciate the forthrightness.

  2. John Zelson says:

    I have to add to the other comments about your honesty in blog: That’s really impressive and I appreciate it. Makes me especially glad i decided to sign-up to bluehost several months ago. Thanks so much!

  3. Thank you for being open. Much appreceiated! I am sure you will get through this.

    Regards
    Jouke

  4. Randy says:

    I am 100% behind Bluehost. I am nothing but delighted with your service. Period.

  5. David Wiseman says:

    Matt,
    I echo other’s in appreciation of the honesty about the situation. I’ve experienced tons of lousy hosts with bad reliability, but none has communicated or admitted there were system-wide issues.

    About the router issue: I’m not an expert on large hosting facilities, but this discussion of “a new router” leaves me less than assured. It reminds me of an outage a couple months back in which it was explained that a power outage occurred & the backup generators were depleted. My feeling was how weak the backup allowance was.. like your capabilities were exhausted in a few hours instead of being covered for a day or as long as it took. I thought things like: how about more diesel delivery or bigger tanks for the generators. So now we are expect to get excited about implementation of one new router. I refer clients to your service, and I’m aware that you are now hosting hundreds of thousands of domains. Are we expected to take comfort that there is ONE NEW ROUTER? How about redundancy? Also I recall another outage a few months back when the problem was reported that your ISP was down. I’ve seen many other hosts with redundant connections to the backbone. Can we not have at least one fallback position since you have the buying power of hundreds of thousands of customers? Maybe I’m misunderstanding the situation, but if I have I’m sure others have too. Perhaps you can clarify your preparedness on these topics:
    -routers and other critical system-wide affecting hardware.
    -backup generators and what your contingency can handle
    -connection to the internet backbone, and how we are protected from one lame ISP’s service or equipment taking us down?

    If your coverage in these areas is as weak as it seems I wonder why I don’t fire up a couple servers in my basement & use lame qwest connectivity (or better yet affordable speakeasy with specified service guarantees and response time guarantees). I can afford one router and a generator and gas to go more than a few hours.

    Don’t get me wrong. I love your company because of the value and mainly the capable techs who answer the phone. I’m suggesting that since you’ve opened the honesty channel with us, please clarify what levels of contigency you have in place for items like those listed above.

  6. David says:

    Matt,

    August has been aweful for spam … what is happening???? Why can I not reach you???

  7. Eric Gooch says:

    Thanks for the updates, Matt. Every company has problems from time to time. What annoys me is when you never hear from a company what’s going on or why it’s happened.

    Despite these glitches you’ve talked about, I’ve been happy with BlueHost. The updates make a difference, and make me feel like I’m doing business with a company that wants to keep their customers “in the loop” both good and bad.

    Thanks.
    -E

  8. I confess that I have had second thoughts about your service over the past few weeks, not least because I recommended you to several people and … well, you know how that feels.

    While your candor is welcome and refreshing, I do think more proactive thought has to be dedicated to capacity planning and redundancy issues in an environment where your success can at times be your greatest enemy. Earlier there was the denial of service issue because of bad guys – something you fixed with better gear – now this – also fixable easily it seems.

    Can I suuggest you think around these problems and ask the simple question “What else should we be doing to preserve a secure and reliable environment assuming continued growth?”

    Thank you.

  9. Brian says:

    I agree that a status page of some sort would alleviate some of the angst we have when the servers are not functioning correctly and we don’t know why. This is exactly the time when Bluehost doesn’t want to be fielding “what’s up?” phone calls. I left my previous two hosting companies for precisely this reason — no feedback.

    Keep up the good work!

  10. Manish says:

    Thanks for being honest matt. thanks for your effors

  11. Gabriel says:

    Hi, I’m BlueHost costumer from Brazil and I’m really happy with your service. Congratulations for your services and for your honesty.

    Keep up this great service!

  12. Don says:

    Stick with Cisco products :-)

  13. Hey Matt ,
    I am proud that I switched hosters I went round and round with the GM of the previous company that i was with cost me thousands of $$$ and he could care less. You just proved to me that Blue Host is well deserving of my business. I will recommend to all my clients that they use You. Im glad to see the CEO of the company take responceiblity for the troubles of the company instead of pushing it off on someone else. And not getting things fixed. You explained what was going on and how you were fixing the problem. Thats how you can tell your a good company not afraid to tell it how it is. Lots of companys dont do that they just hide there problems.
    anyways , Im glad i switched you guys are the best Hosters Ive used and i have tried few . love the plans you offer and the space. you pay a little more but the Quality is there.
    thanks for your Honesty
    Scott Chenoweth /Owner/CEO
    Clear Image Computers Inc.
    Performance Computers Plus LLC.

  14. Thank you so much for all of the work both you and those who work with you do each and every day. As an IT person myself, I can appreciate those days when nothing seems to go as you would like.

    Personally, my best day is when my phone never rings! The side effect, is that no one ever says anything when they don’t have problems. I have been a customer for two years now and have just renewed for another two. Because of the hard work the staff of BlueHost does behind the scenes, you made my choice to stay very easy.

    Thank you so much for the work and effort you put into providing this service!

    Brian

  15. Hey Matt cool blog
    Thanks for being Honest, there really isnt anything we can do for outages or equipment that doesnt work, but its cool to know that we get an honest answer, and guess what all you guys and gals are in the same country as your servers, not one in California and the techs in INDIA!!!
    I got together with you guys a while ago as my ISP (so they call themselves) was useless (understatement)
    But you guys are and where honest with me and always helped me.
    My members now say how much faster and better it is to get the downloads from my site and look at the Gallery for the pictures of last weeks rides.
    Thanks again

    MÀÐÐØGMîÇK

  16. b says:

    Thanks for your honesty. I’ve heard good things about bluehost, and I’m planning on switching as soon as you’ve worked through your current problems. Be sure to keep us informed, so we know when the storm has blown over.

  17. Revolutionary – telling clients the truth!
    We appreciate it.

  18. jrwills says:

    Matt, I’ve just signed up for a two year gig with bluehost. I suppose reading about all these problems should scare me a little. They don’t.

    Years ago I had a boss who had to call a customer and explain why we hadn’t sent parts that were keeping a large piece of money-making equipment out of production. Our only excuse was that we blew it and forgot to order it.

    What was he going to tell the customer? I suggested that he might try the truth. We blew it and would pay to overnight it to him asap. Boss was happily surprised when the customer said, “I appreciate your honesty and the fact that you are resolving the problem as quickly as possible. Things happen in business over which we have no control. I can deal with the truth.”

    Happy to see that’s the way you do business here. Thanks for being upfront and not making excuses. Been down that web-host road! :-)

  19. Benji says:

    well to be honest with all of you that dont like bluehost.com all of you are wrong.. dont you lot make mistakes on real life for something??? this is the same its not bluehost.com they are trying to give as the best benefits that they can, i know that we are paying for it, but looking at other webhosting companies that i been before Bluehost.com Rocks comprare to globat and 1and1 goddady etc i’v had nearly 7 hoting accounts with different companies but non of them satisfied me and all i got from them excussed and nonsens emails and support..

    with bluehost.com you get the support imediately email, phone or the new chat support that they provide now.

    bluehost.com is the best hosting company that i have used till now as i said i have had 7 hosting accounts with different companies but non of them were to my satisfaction..

    Now once again for those that arent happy with bluehost.com you dont need to blame matt coz his trying to give us the best, and i really love being with bluehost.com and i will carry one being with them they rock.

    Bluehost.com allways the best.

    Benji

  20. Lee says:

    Thanks for the lesson on how to handle yourself when things don’t work out quite the way you planned them. Come clean, and get to work making it right. Sounds so simple… Bravo! I’ve been with Bluehost a while now and must admit, “You had me at hello.”

  21. Dirk says:

    Thanks for all of the updates. I have not experienced any issues but I still am appreciative of the notifications. My old hosting provider wouldn’t tell me anything even with all the down time! You guys beat them hands down!

  22. Bob Mutch says:

    Matt,

    Slow slow slow, my site is slow,
    I don’t know what to do, I don’t know where to go,
    Advertisers is call’n, sales is a fall’n,
    Confidence is a fall’n, tech support I keep call’n.

    Then I find a blog, dat tells me the truth,
    It don’t hold nothing back, just lets it all loose,
    Go’n to Cisco, is the right move,
    Huck’n A-D-F, you’s hitting the grove.

    Get’n room, in the right place,
    More U-P-S juice, is less disgrace,
    Roll-your-own kernel, the less you crash,
    Reliability’s up, but you’s make’n less cash.

    I’m will’n to stick, for another while,
    Due to your blog, it kind of made me smile,
    Cus dats the way, I would do the thing,
    Tell’n the truth, has the right ring!

    bobmutch

  23. chad says:

    I’ve been very impressed with Bluehost so far (support especially). I sing your praise daily to a number of tech people in my area. Please keep up the good work.

  24. Kina Fink says:

    I know you say bluehost is having problems, but me and my boss have been sooo happy with your service. Your customer support has been over the top excellent and I feel that we get some great extras with the hosting package. Keep up the good work!

  25. Matt,
    I have been growing fond of Bluehost, I have several websites under construction and migrating over to your company servers…but, after reading your ‘confession’ note, I am now a full fledge fan. It ain’t easy to be forthcoming in business. I admire your personal touch. The honesty and warmth mean a lot to me.
    I will be a customer for a long time. In fact, I just realized that I could be an affiliate and earn big bucks as well. So the last 5 clients I sent to you could have bought me dinner…in Paris. But, hey, they are happy with the service and I am happy to spread to the word.
    With a service backbone like yours, I am encouraged to build more websites.

    Unlike ‘other’ host companies I have used in the past, your sales reps did NOT try to hawk me into buying more stuff I did not need. Bravo!!
    I’ll be buying plenty more as the days roll out anyway.
    I am just about to post 3 more sites this week. http://www.xnews.com
    http://www.EinsteinsMind.com, http://www.eTrips.com, http://www.eyefullTower.com
    At least 4 more in October.
    Anyway, thanks again for your amazing and refreshing approach to business. I wish you every success.
    PS I loved the lemonade stand story. Way cool in the family realm. Way cool!
    Cheers,
    Lee Mclaughlin
    photography ~ film

  26. R says:

    You clearly took the right approach to discussing these issues. Te majority of your clients responding are happy with the explaination and honesty. The email you sent today also helped you save a few clients. I know because I have seen at least one decide to give you a second shot today. But looking at this as another person in the technology business why didn’t you guys plan ahead?

    Growth should be anticipated, if an old router is starting to give you problems it’s time to chuck it. Better yet you should have been able to anticipate that it would no longer be an appropriate appliance to support your users by x-date and retired it before then. Although I am not on the tech side of the company I know that the supervisors on the tech team need to be business minded or be made aware of objectives by someone who is. Outages happen, technology is finicky but two months is an awful long time to spend on one issue. Especially with a full time technical staff. In the mean time you have clients losing money. I’m not sure you have your priorities in the right place here.

  27. Brian says:

    I just received an email from you regarding the issues, and the first thing I thought was how nice it is to have my hosting company actually contact me when there is problems, instead of me finding out about a problem and trying to wrestle with a support team to find out what happened. I really appreciated this proactive approach (which I definitely NEVER received at my previous hosting company). Apparently every one else appreciates the honesty and communication as well!!! Keep up the great work!

  28. michael says:

    Thanks for the prompt work – and all cred to you for your honesty and resolve. It was only the last couple of days that I have been noticing a slow down.

    Thanks

  29. Brian says:

    I’m beyond impressed with Bluehost’s responsiveness to recent outages. This is my first year as a customer, and I can guarantee that due to your service and support, it won’t be my last!

  30. Eric C says:

    As a user for the last year and a half, I can’t tell you how refreshing it is to see someone actually “tell it like it is” with regards to their business. It’s one of the reasons I have stuck with Bluehost (as well as all the great features, for practically nothing per month).

    Also, would love to help out with regards to your new facility. I have a vested interest in keeping Bluehost up and running, and as I work for APC, maybe there is some expertise I could offer with regards to your move, sizing the UPS and generator, etc….

    Keep up the good work!

    E

  31. Simon Kirk says:

    I originally signed up to you guys while trying to find a nice cheap hosting company that offered everything I needed to run a few websites off. Since then I no longer use my account as heavily – but to this day its still the only account I use, I have been using you guys for about 3 years.. and sure there have been problems off and on.. but Im so impressed with your honesty, and loyalty to customers, and the fact that I know your all slaving away to get something sorted, that I do not have a problem with your service at all. I still recommend you guys to everyone I know, and I am very pleased with how things are!

    So this may not be overly relevant to your blog, but I’d just like to say a big thanks to you guys for doing an excellent job! I look after approximately 500 servers and 10,000+ computers for a small part of the NHS in the UK, and I know exactly what it can be like with people screaming at you for fixes when things go wrong, and how these things can never really be helped!

    Thanks again for your honesty and in my opinion, excellent service!

    Cheers,

    Si

  32. Matt,

    Thank you for providing a window into your business feelings and expectations. Your honesty and selflesness that your writing portrays is both refreshing and novel. It is easy to get pissed and irrate at a corporation that shows no humanity, a huge entity that cares less about you and your dissappointments and more about the almighty dollar. Hearing the situation in your words, getting a sense of the frustration that not only you, but your customers feel, really allows peopleare to show a patience and tollerance that most businesses in america don’t get nor care if they recieve. Thank you for making this business personal…it ads a very positive and human side to business and it’s leaders.

    I am enlightened

    Thanks,

    Dustin R. Sponsler, M.Ed.

  33. PoRtCuLLiS says:

    Thanks for bringing out the new ‘High CPU’ plan, I have high hopes that it will end my problems.

  34. Stuart Dallas says:

    I appreciate the honesty sentiment – in fact I applaud it. However, is this really honesty or a response to a growing number of complaints dressed up as an honest CEO? Let’s review the first sentence from your post on August 4th…

    “Things are going very well at Bluehost right now” [http://mattheaton.com/?p=49]

    Judge for yourself… Is Matt really being honest or has it come to the point where nobody will buy that everything is peachy so this post was created as an attempt to explain away the cracks in the service?

    I was considering getting an account until I read this blog. David Wiseman hit the nail right on the head (http://mattheaton.com/?p=51#comment-9487) – a professional web host worth paying for should not be knocked out by power outages and faulty routers. Sorry, but not good enough for me.

  35. Bob Mutch says:

    Things are getting worse on PHP5 servers I have been patience with bluehost.com but that is all over. They just can’t get there act together. Matt you are running your linux boxes like they are windoze boxes. Two or more times a week for the last month I am down for extended time. I am out of here!!! I hope you get your act together, people will only put up with poor service so long.

    Also I don’t think you will post this.

  36. […] So, I would not recommend bluehost.com until they get their act togehter. I am sure they work very hard in this cut throat business. Combined with their recent service issues recently, I would give them a bit of time and check back later to see if they improve. […]

  37. After experiencing horrible downtime problems with BlueHost I decided to search the issue to see how widespread it is. I hadn’t been able to find any answers on the BlueHost site. I AM glad to have found this blog and I appreciate the frankness, however it is a disservice to your customers not to publish this information (or at least a note describing prolems) on the BlueHost site.

    I am a web developer and recently switched my company’s site to BlueHost. I had heard that BlueHost was A+ so I assured my boss that the service would be great. We took the site off of a dedicated server to save some money. Now, I’m afraid that the service we’ve been getting is making me look very bad. I’ve been made a lier and am not happy about it. I hope that the problems can be fixed soon.

    Also, just FYI but your phone system needs some work. I’ve memorized your 5 Michael Hedges and Enya songs and have been routed to a busy signal several times after waiting 30 minutes for help. BlueHost has very affordable plans, but I won’t be switching any more sites over until things are better. I will hold on for a little longer but my patience is waining.

  38. […] Ditto for BlueHost, where my main blog is hosted. They had a gaggle of problems a while back. But there was no evasion. Their CEO posted on his blog what went wrong, apologized, then explained how they’ve fixed it. Which they did, and the blog has been solid as a rock ever since. […]

  39. Brouhaha says:

    One thing that amazes me is the number of DDoS attacks that are directed at you… why is that, do you think? Our sites are down again today due to an attack. Seems like a few times a month! Isnt’ there more you can do to deal with this? I expect people to attack big well-known consumer sites, but why Bluehost? I hope you are working with law enforcement to identify who is attacking all the time.

  40. […] The best part, in my opinion, is that the CEO takes the lead in customer service! He has his own blog, and he’s pretty darn frank about the good and bad of his company (note that his blog talks about Bluehost, a different hosting company but Hostmonster is actually a sister company of Bluehost, and they use the same physical infrastructure). If you don’t believe me, check out this post, where he was completely upfront about the serious server problems that occured last couple of months. After reading it, I feel completely at ease with being their customer. I don’t think that it is likely that they will just disappear without a word. Great customer service culture, and certainly goes a long way to keep their existing customers! […]

  41. Thank you for your honesty. When I notice our site is down I worry how many times our site is down without knowing it. This honesty helps, but is there any way of providing a report that I can use to look up any service interuptions?

  42. Frank says:

    I have used bluehost.com recently with one of my sites, just to see if it has changed. Things seem to be running a lot more smoothly than before!

  43. Jason says:

    Bluehost is good for the money, provide a lot for less, and they have a solid network. Our Independent Review panel has seen better support from other companies in it’s class, but for the value, Bluehost PHP solutions are very good.

  44. Brain says:

    Matt,
    I really appricate it

  45. Technoboy says:

    wow! I’m just shopping around for a hosting service to demo my recently acquired PHP skills, and this post just might be the deciding factor for me to pick BlueHost!

  46. michead says:

    Great! Matt, I am clapping you.

Leave a Reply