mattheaton.com

As I think back over the last 8 years, while I sit in my cubby in my bedroom, a big smile comes to my face. I can’t even begin to mention all the crazy things that have gone on in the Bluehost offices. Bluehost grew from a 4 person company to a 300 person company over the last eight years . Yet, somehow we managed to keep it fun and stay reasonably focused on just one thing – Shared Web Hosting.

It was the singularity of that goal to provide the best shared web hosting that we could that was the most important ingredient to our success. We never tried to be everything to everybody and that is why it worked. When competitors saw other opportunities and took their eye off the ball we stayed focused.

As I look to the future of where Bluehost needs to go and how it will get there I don’t know that I can be the one to continue to provide that kind of focus. I am 100% a techie through and though. Being a techie means that I am distracted by shiny new things Lately, I have had a lot of interest in lower level linux kernel system development. I have also built a data center that is now operational. In addition, for whatever reason I have become obsessed with building out/trading dark fibre to create my own nationwide and worldwide fibre network.

While the pursuit of all of those businesses can greatly benefit Bluehost, Hostmonster, and Fastdomain with lower costs, improved speeds, and greater reliability those goals also can’t coexist with my responsibility as CEO of Bluehost and Hostmonster.

To that end, this blog post will serve as my public resignation as CEO of Bluehost Inc and Hostmonster Inc. I am assigning myself to become a system architect within the company and will continue to work on the design and technical structure of our hosting platform. Dan Handy, our current COO, will take my place as CEO of Bluehost Inc and Hostmonster Inc. In reality Dan Handy has already been filling this responsibility for the past several months. This announcement is simply formalizing an already existing situation. Dan Handy is great and Bluehost is very lucky to have him. He brings structure, organization, and order from the chaos that tends to surround me. I can’t thank him enough for that!

I love Bluehost. I’ve poured my heart and soul into this company. I won’t let it fail. To this end, I have reserved enough control that when a few heads need to be knocked around to get important things done that I can still be the one doing the knocking if need be.

Lastly, I want to offer my sincere thanks to all of our customers. Sometimes we did a great job, sometimes we fell apart, but you stuck with us through all of it and made us one of the most successful hosting companies on the planet. You are the reason I get 4 hours of sleep each night. I hate the thought of disappointing you, and have always tried to do my best to live up to your expectations. THANK YOU!!!

This blog will continue on so that I can do some shameless marketing for some of my new ventures and continue to ramble on about technological changes and innovations, but I will change the link on our sites to read, “Founder’s Blog”, instead of “CEO Blog”. So if for some strange reason you have desire to visit again there will be some new content Thank you all. It was a privilege and an honor.

Thanks,
Matt Heaton / Founder Bluehost.com / Hostmonster.com / Fastdomain.com

Its been many months since I last wrote on my blog – I apologize for that. Its been a very busy few months, but perhaps I can settle into a routine and write a little more often. I am writing today to finally announce that our new data center is up and going! Its the first data center that I have built and we did *MANY* things that are very nonstandard as far as cooling and power management. It really is a great to see it finally come to fruition and get some use. I thought you might be interested in a few facts about our new data center.

– Approximately 12,000 sq ft in size
– 4.7 Megawatts of power available for use
– 720 tons of cooling capacity installed (Can you say windtunnel!!)
– Actually uses 100% outside air cooling for about 7.5 months per year (At significant power savings!!!)
– 20488 U of total server space – (Total of 394 8′ cabinets full of computer equip)
– 300,000 CFM of heat exhaustion out the roof (More important to get hot air out than cold air in!)
– 2x 2 Megawatt generators (Room available to add a 3rd 2 megawatt generator)
– 4x 750 KW UPS (Plus an addition pair on the outer edge if we end up needing it)
– Has capacity to deliver up to 15 KW of power per cabinet with proper cooling ready to go
– 36″ raised floor across the entire datacenter
– 4 separate physical fibre lines brought directly into building through two diverse entrances with access to all major IP/transit providers
– Keycard/Hand scanner access for each hot aisle and each cold aisle as well as cameras for each aisle
– Complete isolation of hot and cold aisles
– Variable on the fly settings/tunables for exhaust fans, supply fans, heat exchanges (To heat up the air when outside air is too cold), chiller pumps, CRAC units. Everything literally adjusts itself every 30 seconds automatically so that you never pay for too much cooling and never cool air when its less expensive to use outside air. Its really an AMAZING system!
– $8.5 million to build (An existing outer building) – That is actually really inexpensive for the capacity/features we got out of it! Thanks to Noel for getting the price so low and under budget! The engineers told us it would be impossible to build for the price we wanted – to them I say

I really need to go in and take some pictures to show how it works. I know most of you just want your hosting to work better and I promise this new data center will help uptimes be better!

The data center is actually not owned by Bluehost/Hostmonster. I built this to be a separate business entity completely. Bluehost/Hostmonster actually only take 25% of the total capacity of this data center. I guess what I am saying is that if you need data center space give me a call! We have 15,000 U available right now in a brand new data center. We also sell bandwidth for very very inexpensive prices and never make you sign a long term contract. You can buy 500 megs or 20 gigs of bandwidth a month with no commitment at all on bandwidth or term. Who else does that?

Now that its done I find myself wondering how big the next one will be? If this one goes well I would like to build a 75,000 ft facility. I think I already found the land for it

Thanks,
Matt Heaton / Founder Bluehost.com/Hostmonster.com

At Bluehost/Hostmonster we have been developing some really neat tools to segregate cpu, disk I/O, and memory. Here is a link to one of my blog posts in the past that describes what we have worked on in detail.

Up to now we haven’t really been able to sell this product because it was simply too difficult to make it work in most company environments. It required a custom kernel patch, significant system knowledge, and really just required too much effort on our part to make it work well in other scenarios other than our own. However, those days are quickly coming to an end.

Now that Redhat has finally released Red Hat Enterprise Linux 6.0 we have a “stable” kernel with which to release kernel RPMs. Redhat, and most other server/enterprise Linux distributions (Such as SUSE) have settled on 2.6.32 as their baseline kernel for their server product offerings. This is a very workable kernel for our feature set and allows us to make implementing our cpu, i/o, and memory management tools extremely easy.

We currently have a backlog of several hundred companies that have expressed interest in these products. I am sorry it has taken so long, but it was absolutely necessary that we had a standardized kernel newer than 2.6.30.

To be very clear, in order to use our products you will need to be running RHEL (Redhat Enterprise Linux) 6.0 or greater or SUSE Enterprise Linux 11 or newer.

We will most likely set up our own repository for these products as well as a free respin of the RHEL 6.x distribution compiled from source minus the Redhat trademarks (Think CentOS without the 2-3 week delay that often accompanies new releases). The 6.x distribution will be free for anyone and everyone as well as any updates, etc. I am willing to provide whatever bandwidth, servers and personnel are necessary to keep us within 24 hours of all of Redhat’s updates.

Thanks to those that have waited patiently, as well as those that were a little agitated We will very soon have a product for you to use!

Matt Heaton / Bluehost.com

WordCamp Utah is coming up this weekend Saturday August 28th from 9am-5pm. For more information about the event visit: http://2010.utah.wordcamp.org. BlueHost does not normally sponsor events or setup booths at tradeshows mainly because its too much of a hassle just to try and get your name out there WordPress on the other hand is one of our favorite and most popular open source products available. Over 650,000 WordPress blogs are hosted on our servers and that number grows by hundreds per day.

The keynote speaker at this event is Matt Mullenweg who is the founding developer of WordPress. There are also a number of great learning sessions that can help you get started or enhance your current web presence using this amazing FREE tool. See the full schedule at: http://2010.utah.wordcamp.org/schedule/ for more information on the speakers and sessions that will be available.

Also, BlueHost is buying lunch for EVERY attendee, and we want that bill to be HUGE. So if you haven’t signed up for the event yet go register now at: http://wordcamputah2010.eventbrite.com/ and make sure to stop by and say “Hi!”. I’d love to meet some Bluehost customers in person!

Thanks,
Matt Heaton

Its Hostingcon time again this year. This year its in Austin, Texas. Its nice to see all the email addresses actually
have a real person associated with them

Every year that I attend I run into Jack from Parallels. Jack is Parallels #2 guy. A few years back I challenged
Jack to a pingpong match. Now, I think that I am pretty good at pingpong, but Jack beat me by 2 points in the 5th
game a few years back. I am now trying to use my blog to get Jack to play one more match! So what will it be
Jack? A rematch If you beat me I will get up publicly say that I think Parallels way of doing things is better than
Bluehost!! What could be better than that? Don’t tell me a Parallels guy is scared of little ole Bluehost

Matt

Hello again! Its been quite a while since I have written and I thought its about time I got around to updating you on some of the new things we have been working on lately.

Some of you may know that in addition to shared hosting I have several other small/big projects that I like to work on in my spare (haha) time. I’d like to tell you quickly about some of those and then briefly mention a few changes to Bluehost and perhaps a couple of interesting stats about our business.

1st – Fibre Network – I have been working hard trying to piece together our own nationwide (Soon worldwide) fibre network. This becomes an integral part of some of my other projects that I am investing my time in. We will have a robust peering network with many tier 2 participants and hopefully soon (With enough bullying on my part) a tier 1 provider or two! The first part of this project linking Los Angeles to New York will be live by the end of this month (Hurray!!)

2) SUPER COOL caching service (Think CDN – Content distribution network). This one is kind of hard to explain, even to those in the business. We have a fundamentally different way of implementing internet edge caching servers that I think will change completely how the game is played. Look for this stuff to be out in about 6-7 months. Akamai – I know you have a $7.5 billion market cap, but our technology is better Keep a close watch on what we have coming out. We are gunning for all your business!

3) CPUD – I/O Throttled: These two technologies are what allow us to provide VPS like services in a shared hosting environment. They will work on any linux box with a 2.6.32 kernel and above. They were developed almost exclusively in house and work extremely well. We haven’t sold this to any company because of the difficulty in making it work with older kernels, but all that will change when Red Hat releases version 6.0 of the RHEL distribution. Once Red Hat releases that version we will offer a simply RPM install of our product and everyone will finally be able to purchase this product! Sorry to the over 125 companies on the waiting list for this product – It will be out soon!

4) New Premium Data Center – We have been working on our new data center for a while now. It is scheduled to be done in December, but its looking like January 2011 will probably be the real date. Its not huge by data center standards, but it will give us 5 megawatts of power for 400 cabinets (47U each). It uses a VERY nonstandard cooling system designed by us in house which will save us a ton of money on cooling costs, and is located right in our office building where all our employees are located. This ensures best quality service for all our customers.

5) Quick Backup – we have a very cool backup product that is also developed in house. It uses custom kernel mods to enable an extremely fast backup process for our Cpanel customer servers. What used to take us 10 hours to accomplish now takes about 45 minutes. It is a complete drop in replacement for Cpanel’s backup system that is totally compatible with their restore process, but uses about 1/3 the space that normal Cpanel requires. We are using it live right now on about 60% of our servers and only need some additional hardware to put it in place in 100% of our network (Happening right now).

6) I guess I should mention that the old Bluehost.com main site has FINALLY been replaced with something decent. It was never that great to begin with and had morphed over time into a complete mess. That being said, it had great search engine rankings and helped us sign up more than 575,000 accounts! But alas, I am sad to see it go Its been like an old friend these past 7 years!

7) Oh, I forgot to mention this. I wanted to announce when we finally hit 2 million domains hosted. I got lazy and never mentioned it, but now we are almost at 2.2 million domains. So lets this be my official announcement that we are now *officially* over 2.15 million domains hosted! Yeah!

We are now (I guess we have been for awhile) the 5th fastest growing registrar in the world. For those that are wondering what a registrar is it is an ICANN accredited company that sells domain names. Whats interesting about this is that you can’t even purchase a domain name from us unless you are a hosting customer. So to be the 5th fastest growing is quite an achievement given the requirements we place on the customer.

9) We are now the 10th largest hosting company in the world. Yeah! And 2nd or 1st fastest growing hosting company.

10) We have now surpassed Yahoo in domains hosted and hosting accounts. I’ve heard of Yahoo? I think they are one of the bigger internet companies

11) We FINALLY got a very nice front desk in our main office. Long live the plastic table that we used for the past 18 months

There you have it. Those are all my significant technology related activities as well as a few interesting stats about Bluehost. Thanks to everyone out there that has made Bluehost/Hostmonster/Fastdomain possible. We appreciate all you do for us and hope to live up to your expectations as a hosting company!

Thanks,
Matt Heaton / President Bluehost.com

As many of you know I haven’t been the biggest Microsoft supporter Its been a really bumpy road using Microsoft products over the past 20 years. The consistently buggy software, and the security nightmare that is Windows/Internet Explorer is so flawed that I don’t think its possible to “fix” at this point without a fundamental change to the lowest levels of the OS.

About 4 years ago I switched all my desktop use to Mac OSX. I love it. And although I believe it is superior to Windows in almost every way (Especially the MACH Kernel) I didn’t really care what we used at Bluehost/Hostmonster/Fastdomain for our office/support computers.

Those days of ignorant bliss are now totally gone. We literally can no longer tolerate Windows in our corporate network. What you say? I know this sounds strange as corporate America is a stalwart Windows using market, but I’ll say it again. We can’t tolerate Windows anymore. It is simply too high a risk. It is UNSECURABLE.

Now I know right now that there are going to be lots of readers that write back saying, “If you know what you are doing then Windows is easy to secure. Only people that aren’t technical enough to do things right get infected.” Uh huh. There are so many zero day vulnerabilities with Windows that I simply can’t keep our network secure. In addition, because of the type of business we are in we are constantly the target of malicious activity directed at our network. We have to protect ourselves, and running Windows feels like bringing a plastic knife to gun fight when it comes to security.

So now the vast majority of employees at Bluehost/Hostmonster/Fastdomain run Linux in an LDAP environment (Support, Abuse, Admins, etc). We have several Macs throughout the office and will be adding many more in the near future. We do have some Windows machines (Quarantined from the rest of our network completely) that are used for testing, and a couple that are still used for accounting purposes, but even most of those are just virtual machines running Windows.

I’m not saying Linux, and Mac OSX aren’t susceptible to viruses or malware, but its so much less that its not even close. Its strange to me that most businesses just accept the fact that their machines will be infected with Malware and that they will just clean it off with the latest scan of their computer. Are you ok with your passwords, personal data, and financial information just flowing out to owners of various botnets? Are you fine to have an army of your office machines sending out spam in the background all day long while you wonder why your internet connection is slow? Clearly the answer for most businesses is yes. I guess after 20 years of Microsoft usage most people are just beaten into submission that computing has to be this way – but it doesn’t have to be. There are alternatives, and we are using them. I hope many of you reading today will consider some of the alternatives.

Matt Heaton

We have worked so much lately on many internals to our hosting platform. Sometimes I worry that our customers think we are standing still when they don’t see a lot of outward/customer facing changes. However, we have been feverishly working on improving the overall stability and speed of our platform.

We have nailed down CPU overages, memory overages, and disk i/o bottlenecks (i/o was the hardest one btw). We have delved deep into the dark art of linux process scheduling and have come away mostly unscathed

All of this backend work has made a tremendous difference to our customers, but now we have something new to announce. MySQL process scheduling protection. I know what you are thinking… That doesn’t sound exciting at all, but to us computer nerds its a big deal!

Basically a single rouge php/perl/ruby script utilizing MySQL can consume all the resources on a server and never even show up as using much CPU at all. I have written test scripts that consume less than 1% of the total “cpu time” yet leave a machine with 16 cores with 0% idle time on all cpu/cores and thousands of backed up processes. I can replicate this on virtually every hosting company that I have test accounts on (And I have a LOT of test accounts on competitor’s servers). This isn’t a rare thing that happens. Its VERY common, and many popular plugins for WordPress and phpBB cause this to happen very frequently.

Surely something like this wouldn’t be allowed to happen in a mature multiuser environment such as Linux! When I first discovered this bug I didn’t think it was wide spread. After running MANY tests and spending a lot of near sleepless nights proving my theories I discovered that it was happening hundreds of times per hour for short periods of time across all our servers. This causes short delays (Usually 5-60 seconds) that are very very difficult to track down after the fact.

The issue really is with MySQL itself, but it also is because of a serious design flaw in the current linux process scheduler. I am intentionally being vague on the specifics of the problem because I feel the fix we have developed will give us a substantial competitive advantage. Without question we will at some point in the future release this code to the community, but for now we will continue to use it in house to the benefit of our loyal customer base.

This speed and stability fix will impact directly (In a negative way – meaning that coding problems will have to be fixed on the customers side before their sites become usable) about 400 customers out of 547,000 (At the time of this blog post). Indirectly, everyone else will benefit greatly because it literally makes the difference for many of our servers from being 0% idle (Totally overworked) and 50% idle.

It is not yet live on our production environment (Except in a few controlled cases), but will be completely live on our system by approximately Wednesday of next week (March 31st, 2010).

Don’t expect a huge change, but certainly expect and demand stability from us as your hosting provider. Just remember that when you don’t have slowdowns and problems it’s NOT by chance. We kill ourselves trying to make Bluehost/Hostmonster the best shared hosting on the planet. We think we have succeeded, but if you don’t think so then the job we are doing isn’t good enough. Either way we will continue to do our best to solve the biggest issues as we see them. Thanks again for your business and your encouragement.

Matt Heaton / President Bluehost.com

Bluehost/Hostmonster/Fastdomain started very small. We never took a dime of investment capital, never had any debt, and were happy to wait the 18 months it took before we got our first paycheck.

This may sound quite risky to many of you out there, but for me it wasn’t at all. I could see in my mind completely the plan for success. I defined success differently back then and I honestly never planned for this business to grow this large, but I had a clear picture of exactly the steps required to succeed. In my mind it was just a matter of doing it. I never thought there was a chance that it wouldn’t succeed. This is the how an entrepreneur thinks. They solve problems, take risks, work hard, and have an insatiable desire to succeed.

Now that we have grown into a much larger company we have somewhat outgrown the stage where only an entrepreneur is needed. Now we need an entrepreneur and a great leader.

This got me thinking what the difference is. I wanted to share with you what I believe the difference is. Here is the Matt Heaton definition of each.

Successful Entrepreneur – A person who has the ability to recognize a need/deficiency, ability to differentiate between a useful need and an idea that can be a successful business, design a solution, use his/her drive and ambition to implement the solution , and then profit from that solution to the desired level of the entrepreneur.

Most successful entrepreneurs follow this path reasonably close in my opinion. The unsuccessful ones are nearly identical in almost every way to the very successful entrepreneurs except for two missing attributes. If they lack the knowledge to implement their own ideas themselves they often fail. This happens because sometimes if you rely on someone else, or outside help the ideas tend to change and the vision that they clearly saw at the beginning of their plan begins to fall apart. The second area is intelligence/education. If you have all the ambition in the world but don’t understand finances or your product or the marketplace you will almost certainly fail. I am not talking about a degree or any specific piece of paper. I simply mean that you have to be willing to put the time in to really understand the specifics of the problem you are trying to solve. If you do that you will succeed.

Successful Leader – A person who has the ability to recognize a need/deficiency, ability to differentiate between the most important goals from those that can and should wait, ability to design a solution that can be implemented with the resources he/she has available, ability to obtain currently unavailable resources to achieve the outlined goal, use his/her drive and ambition to implement the goal using the resources and people around him, and then show others how the goal solved the predetermined problem and then clearly state what the next goal is and why.

In essence, for me the main difference between a great entrepreneur and a great leader is how you achieve success. An entrepreneur literally wills his/her idea to come to life and succeed. It all comes from drive and ambition from within themselves. A great leader does the same thing through the people around them. Its easy to make myself be great (Always humble I know ), its MUCH harder to make those around you be great as well.

To be a successful entrepreneur from my point of view is a piece of cake. Its in my DNA, it’s who I am. To be a successful leader is much harder for me. I very much rely on my own abilities to solve many problems at hand. I am often unwilling to listen to others ideas or to give freedom to implement those ideas because they don’t fit within my vision for the business. Sometimes that can be a good thing if I feel the person would make a big mistake, but I have tried very hard to surround myself with intelligent, competent people. If I can’t trust them to do their jobs, then when they fail at those jobs it’s no ones fault but my own.

I’m still deciding if I’m the right person to lead our company in the future. I tend to lead more with a whip in hand then with a kind word and encouragement. Its time for me to decide if I’m willing to bend with the reality of having a large company or break in half from lack of flexibility required to lead a large company. Whatever path I choose I’ll make sure it the best thing for the company, for our customers, and for me.

Thanks,
Matt Heaton / President Bluehost.com / Hostmonster.com / Fastdomain.com

Several months back I took my wife and five children on a 7 day Disney cruise (I *HIGHLY* recommend it by the way, and I’m a hard person to please ). Whenever I go on vacation the first thing I take care of is making sure that I have internet access. Thankfully, I was able to use my Verizon MiFi card while in most ports, but while at sea I had to use Disney’s on board satellite internet. It was extremely slow.

This got me thinking of how I could best increase the internet speed for our clients that have slow internet connections at no cost to them. I decided on using mod_deflate. I had used mod_gzip in the past (Almost 10 years ago) so I was familiar with how it all worked and it was simple to set up. Mod_deflate basically takes certain types of files and compresses them at the server level and then sends those smaller files to you. Images, zip files, etc don’t compress well (And so we don’t compress these, but HTML files, javascript files, css files, etc compress very well. Often we see 80% compression levels on those type of files. These files are then decompressed on the client side automatically and used. This is all transparent to the user, except that download/page load times are much faster for the user (10-25% faster).

However, there is a severe problem with using mod_deflate that no one seems to have solved. Using mod_deflate requires *significant* CPU usage on the server to use. The problem is that often CPU resources are maxed out. If you use mod_deflate while the CPU(s) are maxed out then the servers become even slower and all websites on the server will appear very very sluggish. For this reason most web hosting companies don’t use mod_deflate, and for good reason.

However, at Bluehost/Hostmonster we have a great solution for this problem! Some of you may have read a previous blog post where I mention that Bluehost/Hostmonster have a proprietary CPU protection system. Using, this system we track CPU usage in realtime. We then wrote a patch to the Apache web server (This is what serves your websites to your browser) that interfaces with our CPU protection system. This patch checks our CPU usage twice a second and if CPU usage exceeds a certain threshold then we temporarily suspend mod_deflate. When there are unused CPU cycles then it reenables mod_deflate. By implementing it this way we get all the benefits of mod_deflate with none of the detriments of excessive cpu usage causing slowdowns.

The first full day we ran this it lowered our bandwidth consumption about 600 Mbits a second (With very conservative settings). When we run it with aggressive compression we save over 1 Gig/s of sustained bandwidth. That is considerable savings/speedup for something that took about 4 days to develop, test, and deploy!

Now, next time our family goes on a cruise Bluehost/Hostmoner sites will appear much faster!

Thanks,
Matt Heaton / Hosting by Bluehost.com