When things are good they are great and when they are bad they are REALLY bad. The last 2 months we have been REALLY bad. I know many potential clients read this blog, but I have to be honest with our existing clients that we have been less than great the last 2 months and the problems lie squarely on my shoulders.
Today we had downtime off and on from around 10am-3pm. It was our garbage heap we call a router that did it to us again. We have had a list of things go wrong in trying to get our new router up and going. We have the new router in our datacenter but the fan tray is not the highspeed fan tray we needed and so it is holding us back. We won’t be able to put our new router into production until Friday night at around midnight. It was supposed to be up and running this Monday but evidently next day air means 4 days later to some people.
Here is the short list of majors problems that we have had to deal with in the last 2 months and what we are doing about these problems.
Problem – Major linux kernels problems with Redhat Enterprise 4. This affected about 40 servers. Solution – Finally built our own custom kernel that solved our multitude of issues.
Problem – Two seperate power outages that affected about half of our users. Solution – So far have moved about 80% of our users to a facility that has enough UPS and generators to alleviate the problem. We are signing a lease next week on a 5000 ft data center that will allow us to grow as well as provide colocation services for many businesses. The whole move will take around 45 days to complete.
Problem – Multiple outages affecting all our users. Slow connection speeds and dropped packets. – Solution – A new cisco 6509 router with a sup720-3bxl card to solve our routing issues. This is 95% of the problems that Bluehost has been experiencing. We have been trying to solve the router problem for a couple of months. I think we finally have a handle on this hairy problem!
Problem – APF firewall system running on all users boxes. This firewall software has been randomly blocking data ports for users across all our systems. There was no rhyme or reason to why this was happening and thus very difficult to resolve. Solution – Finally gave up on APF. It is about the worst software I have ever used. We have installed a new software solution on a dozen servers and have been testing it. It seems to have completely fixed our issues. It is being rolled out to all servers on Monday.
All of these issues have made Bluehost’s reliability in the last 2 months completely unacceptable. As you can see from above we know of the issues and have been working nonstop the last several weeks to get things back to how they should be. Thanks for those that have been patient. For those that haven’t been so patient I don’t blame you one bit! We will have things back to normal in a few days.
Thanks,
Matt Heaton / President Bluehost.com
Thanks for being honest!
Appreciate your honesty! Bluehost seems to be a great company and you have a quick positive team.
I am sure that you’ll have things resolved before you know it!
Thank you for posting these issues. I have been requesting this since I signed on at the beginning of summer of no love. Can you expand on this below. Does this mean that my hosting and mail has a new home in the 5000 ft data center or will this require another move in the 45 day window? What down time will you expect? Also your team kept informing me that the router belonged to ELI. I knew this was not the case since they would have redundant routing at their facility here in SLC at 265 E 100 S. Honesty is the best policy here. Thanks.
“Problem – Two seperate power outages that affected about half of our users. Solution – So far have moved about 80% of our users to a facility that has enough UPS and generators to alleviate the problem. We are signing a lease next week on a 5000 ft data center that will allow us to grow as well as provide colocation services for many businesses. The whole move will take around 45 days to complete.”
I recently joined bluehost and experienced downtime the second day. So was getting myself worried about it. But since you’ve told what was the problem I don’t have any complaints
.
Best regards,
Roosevelt Purification
http://www.photoshopcity.com
Oh man, i just shifted server to Bluehost.
But i trust you!
Keep up the great job.
Mr Heaton,
I cant praise you enough for your frank and honest explnation for the, ah, less than optimum performance of BH the last couple of months. Seeing someone being honest and taking responsibility in this day and age is quite refreshing.
Im sure you will be able to rectify all of the problems that you have faced in recent times, and continue to grow this wonderful company.
Matt,
the decision to host with you was sort of a hunch, I didn’t exactly know why I did that, besides the fact that I liked the synchronizity of your site name and mine (bluehost, blueflash).
I did have a bit of frustration when I built the site (being blocked randomly for a few minutes, most likely your APF), but let me tell you something: I have a business and have made a couple of business deals and in these the main factor was always the people I was dealing with. Uptime isn’t anything for me, and being hosted by a company which has a CEO who is as honest and candid as you are really makes me happy
You’re really a man of my liking …
Markus
I want to work for you. I want to help make you faster, stronger, better. Are you hiring?
Matt,
I know what it feels like when things go wrong that shouldn’t. You’re obviously feeling a little stressed about this and that’s good from your companies point of view – it shows you care about the quality of your service.
Keep on caring and don’t let it get you down too much.
Antony
Thanks for being forthright about this. I really does make all the difference. However, the multitude of clients that I have referred to Bluehost as a solid solution are now breathing down MY neck and screaming bloody murder. It’s very possible that we will lose some very lucrative clients over the credibility hit we’ve taken by recommending Bluehost.
The new firewall will be a welcome change. I’ve been having persistent problems connecting to MySQL remotely for the past few weeks because of that so it will be a big relief to see that resolved!
On the positive side, your support staff has been a pleasure to work with. They know what they are doing and are able to get things resolved on the spot. That is a true rarity these days!
Your techs whoa nswer the phone are most courteous and pretty competent and efficient. The headaches that your blog mentions cost me and my team countless hours of anguish. I have been on the phone for hours a day trying to help your techs troubleshoot the problem.
Furthermore, I recently migrated my site to a new box and the implementation was horrendous!
1) I specifcally scheduled that migration for a time that I have low usage….but instead it was at the worst–highest use– time.
2) The databases were not transferred over properly(Half the tables were missing in some databases)
3) Before the migration, Bluehost made no backup of the system…an OBVIOUS protocol to prevent data loss. I was asked if I had a current backup, which I didn’t since the migration was done at the wrong time. I teach Computer Science and the concept of redundancy, preparing for the worst is a very important topic — something not considered important at BlueHost.
4. The firewall exceptions were not transferred.
Each of these problems required hours of troubleshooting. Will all this downtime (and I see it was not only me) I can’t understand why you don’t offer more than just a sheepish apology. A hotel offers a free room when they mess up and perhaps you should consider something like that (maybe increased services/discount)
This is exactly why we have stayed with and continue to recommend Bluehost to anyone who is looking for effective web hosting services.
This is what makes bluehost the best hosting provider out there!
Matt,
Your openness is exactly what I want in a hosting company (along with 5 9′s but you guys are working on that). I love that you don’t think all your customers are idiots. Custom kernels are da bomb!
- Ryan
Dear Matt,
I have been with BlueHost for about 3 years and the service has always been great. But I hate to disagree with you the service has been terrible now for more than 4 months twice as long as you mention. I’m closing most of my clients hosting and moving to 1 and 1. I’m tired of having to give excuses to my clients for you guy’s lack of planning for growth. Also your phone services is really bad I’m on hold and wend it get close the call get dropped and I get a dial tone. Your customer services said that there are not aware of the problem. It happens 3 times just between 1:30 and 2:00 PM today. Also I have been waiting for 3 very important emails since yesterday I call twice yesterday and customer services said there was no problem. Today I talk to Shaw an my 3 email were hang-up on the mail server. If you want more details please do not hesitate to call 916-632-1000 ext 201.
Yours truly,
Rafael C. Leon
Agree that your transparency is much appreciated.
Two questions: 1) Could/should any of these problems have been anticipated with proper development/configuration and testing before putting into a production environment. 2) Without disclosing any competitive information, what order of magnitude of capital are you spending to fix these problems and do we need to be concerned about financial viability.
Nothwithstanding the issues you mentioned, my experience with Bluehost has been excellent so far, and I would recommend you to others.
Thanks.
We appreciate your honesty!
Thanks for being honest about the issues, and for the nitty gritty details. Which firewall did you choose to replace APF?
How is the domain registry service coming along? I would like to transfer my domains from GoDaddy when your competitive registrar service is up and running. Next renewal is in October. Soon?
I’ve been advocating openness in forum and I’m glad you took this step. I hope this continues not only because people want to hear it but because it’s the right thing to do.
Thank you!
Thanks for your honesty about the problems. Your candor is refreshing and is one of the reasons I like Bluehost so much! You treat your customers with respect and have tech personel that can actually answer questions and solve problems! A miracle!
I used to be with Yahoo. If anyone isn’t happy with Bluehost, tell them to go there- then they’ll really have something to complain about.
Matt,
First, THANKS for being honest. Second, THANKS a million for the 30gs of hosting space and the 750gbs of transfer! That is awesomely KOOL. Pesonally, I haven’t experienced that much downtime. I’ve had a few times where the response on FTP was slow. The only “complaint” I have is the page wizard when modifying a web page through the control panel. That does need a “bit” of work, as in a LOT! I know you’re still in beta on that, so patience is a virtue. Do you guys sell that too?
Thanks for being such a handson CEO and having such a great hosting service.
GOD BLESS,
Beti
Matt,
Having previously worked for a web hosting company, I can understand what is going on behind the scenes. However I do think reading what others have said that your company needs to improve when it comes to planning in future growth. It really sounds to me like your techs and sales teams need to meet together more often. Yes both the teams might growl and hate the idea but these things can be discussed at meetings like that. At the same time it seems like your sys admins need a proper kick in the butt because any serious sys admin would’ve known to start out with custom kernels.
Yvo
Hi Matt, Thanks for bein open to the public about these issues.
I Hope Bluehost will also get tons of extra bandwith to cover the (future) growth in customers and monthly data transfer.
Thanks for letting us know, Matt.
And to any new people at bluehost, I want to testify that they normally have outstanding service.
Most budget-priced hosting companies have these problems, in my experience. Yours is perhaps the second that has put a name to the person in charge and been honest about stuff going totally wrong.
I just switched to BlueHost, *after* reading this. I was impressed with the cancellation policy described by your representative as well… very low risk for me.
Good luck, and, please, keep up the honesty and the communication. It’s nice to work with a business that is accountable in this day and age.
[...] CEO Matt Heaton just blogged about it, detailing the various power, linux, router, and firewall malfunctions that resulted in a bouncy past couple of months. More importantly he 1) apologized and didn’t blame anyone else but himself and 2) detailed precisely how they are fixing the problems, indeed, most have already been done. [...]
[...] Referring to corporate blogging, Matt Heaten at Bluehost definitely does “get it.” I’m not a bluehost user (I use dreamhost), but apparently the hosting service has experienced its share of problems over the past two months. Matt blogged about what the issues have been and what he has done to correct them for bluehost customers. [...]
Performance on my WordPress blog is quicker and snappier these past few days, which may well be due to the new router.
Hi Matt
Your open and frank attitude deserves a lot of praise. Problems do happen in every company, but it is obvious to me that you have what it takes to sort them out. I got out of other hosting companies for their lack of transparency, and I am quite happy to have chosen BlueHost. You recently doubled the capacity of our accounts (and just saw the new offer of an amazing 30Gb hosting space!), added many features, and it seems your company is trying hard to stay extremely competitive. Yes there have been some very unfortunate power and equipment failures recently, but by recognizing the situation and reassuring your customers that you are sparing no efforts to fix the problems, you have gained my loyalty and surely that of most customers. thanks for caring about us.
Not sure if it’s related to the recent problems but my sites have started receiving the grey screen of death (“CPU Quota Exceeded”) again. We had this problem some months ago but it seemed to go away when we were moved to a different server, but now it’s back. I’m pretty sure it’s the server’s fault too, my scripts are running quickly (0.05 seconds or under) and the cpu logs show things like this:
Sun Aug 20 19:16:13 2006: used 30.06 seconds of cpu time for HTTP Request: GET /favicon.ico HTTP/1.1
Support seem to be burying their heads in the sand and not responding to the ticket any further
. Some resolution would be appreciated, downtime costs money.
Well Matt,
S**T HAPPENS! Things break, hardware goes bad, ect.. It happens.
I’ve got to say. I’m nothing but impressed with the service I get from BlueHost. I’ve got 2 personal accounts, and I use the hell out of them.
It’s refreshing as hell to be able to pick up a phone, and have a representitive right there. No waiting, no BS’ing.
And the kicker – 30 gigs of storage, and 750 gigs of transfer.. I LOVE IT! Fantastico is just the icing on the cake.
Keep up the good work.
I’ve referred many customers your way. And I’ll continue to do so..
Good to see the honesty and openness, but what I really need is a red light green light on your support site. Then I know Blue host is having issues, and I can work around them. Having to contact support to find out if there is a problem is becoming problematic.
But keep up the good work, I won’t be leaving just yet.
Matt,
Thank you for posting your blog. I was on the verge of moving my hosting to another company as I have been unable to access parts of a new site that I am building over large periods of time. (2 hrs of unavailable time in one day represents very poor reliability)
When you step up and take ownership of an issue as serious as this, it does you a great deal of credit. And as others have written it makes us want to applaud you and stay put to see this glitch out.
Good luck with the work and we look forward to long and reliable hosting with your business.
Mike
How about a status page where we could get updates when things do go South (as apparently they’ve done for box49 today or yesterday).
At least then we wouldn’t be in the dark as to what’s going on to correct the situation.
thanks-
halp
You are honest about the issues, and that’s good that a CEO can admit issues instead of making excuses. I’ve had very rare connection problems, not bad at all. I’ve had hosts that have been down for 1-2 days straight. Thanks for thinking of bluehost’s quality, it’s really paid off. BTW- Thank you Bluehost team for fixing the Paypal payment method.
Hope this 10 minutes downtime will soon be over. Bluehost is the best webhost out there!
Matt,
keep truckin’ with this!!! Thanks for the honestly….customers always prefer to get the straight-up honest truth, rather than some concocted BS.
I’m staying put with BH….Love it!!
keep up the efforts
I am having to move my site away form hostmonster (bluehost’s sister company) because I run a illustration cimmunity website:
http://www.amateurillustrator.com/
…and apparently if my users upload paintings of nudity it is not legal under Utah state law…even though it has been accepted in western art for centuries. i was told that even a painting of a woman in a lake with her back to the painter was not allowed.
So i am moving to GoDaddy where they assure me that my users can upload artwork any artwork they like…although i do not want too graphic work on my site this level of strictness is a joke.
I wish you guys had made it clearer to me when I signed up.
i called yesterday 8/23/06 cause our mailing lists stopped working only to have Option 2 in the phone system not be a valid option (press two for support) DOH!! so i pressed option one for ‘sales’ and turns out the ‘sales guy’ was able to help me just the same!! cool cross training there BH!!
Matt,
You have a lot of integrity and a bold work ethic. Your transparency is admiral, there’s much to be said about opening a door to yourself and bluehost here. You’ll get all that geek stuff working smooth soon…
Stay at the forefront of this blogging thing; keep posting. Tell us about that new little girl of yours, add some flickr pictures. Did you ever come up with a name?
We understand you’re human, and we also realize technology shifts like the mystery of the wind. IMHO you do a great job at rolling with the punches and keeping your jab up. Bluehost is in a class of its own, you’ll overcome!
Matt,
I really hope you get this sorted soon. It’s a week later and I’m still getting down time messages from SiteUptime.
I had to move domains elsewhere, leaving only the hobby stuff on your servers, but the tech support from my new host is really making me wish I could get the uptime from Bluehost to make it worth moving back.
Tony
Thanks Matt. I just signed up for Bluehost yesterday, and I’m happy that Bluehost admits its mistakes and does everything in their power to correct them.
Better luck in the future, and thanks for your honesty
YEAH Salvorix!! You know something good when you find it! Hope your “Blue Host” problems are history for now and that anything in the future is just “jelly beans!” Thanks for another gift of space. NOW I know – no matter what I do (into the future) – - there WILL be room for it! HONESTY – that’s 50% of why I’m here (not to mention room to grow!)
Matt, the route you have taken is very honorable and I commed you and your company on that. I agree with Yvo though. Make sure your techs and admins know linux and networking well. It may have been overlooked in the hiring frenzy. Please don’t neglect security as well. Let us know if you need help.
Keep up the good work!
-A
Its good to see somebody who comes out and explain the situation.Overally I think the Bluehost is quite good despite the recent issues.
Thanks for the update and for the candor.
Your customers are pleased that you care enough to keep us informed.
Things are bound to happen.
Your customer service is outstanding.
The staff is very sincere in their efforts to answer every question and address every concern.
Hats off to Bluehost!
Way to go, CEO!!
Good to see a more complete explanation. Though the power outages were rather poorly handled imo (less than accurate information given out as to what was happening).
What still concerns me is many of these issues are avoidable. Not sure how they got to the point of actually causing problems before they were dealt with. Lack of capital? Lack of planning? Lack of execution? Time will tell.
I renew in 4 weeks. I’m hoping to stay where I am (though as of this posting I’m having trouble logging into my account).
A British expat, I live and work in Kazakhstan. For a number of years I had my domain hosted by a UK company with a good reputation. So too did a a compatriot of mine out here. However, the service became ever more unreliable and this became all the more aggravating because 1) support was only available in UK office hours which was none too convenient with my 6-hour offset from GMT, and 2) only phone number on offer was a premium rate number which cost quite a bit, even with Skype.
Changing is a hassle but I decided the time had come. I can’t remember how our web search led to Bluehost but it did (saw the name in a top ten list then liked the look/sound of your website. We moved the hosting to you – the only problems, which were few, being caused by the obtuseness of the hosting company we were leaving, all easily resolved by calls to your superb 24/7 support line and I have never been happier with my internet.
Today was the first time ever I noticed a problem and used your chat. Once again – instant informed advice from Brett that kernel upgrade is being done and will be over in 15 minutes was all I need to feel okay.
So I, for one, am really happy at Bluehost. For me, the important thing is that I feel things are being looked after and you post here makes me feel better looked after still.
I’ve been web-host shopping for the past couple of weeks. Since it will be a long term commitment, on my part, I wanted to make sure I was picking the hosting company that would be : customer friendly, consistent in their policies and reliable.
All of that is important, but Honesty is by far the most important virtue I look for in dealing with anyone and because of that… I pick BlueHost.
Looking forward to doing business with you.