mattheaton.com

Ok, I’ve done enough personal blog posts that its time to get technical again. Web hosting is a changing animal. Customer needs are becoming more and more demanding as businesses rely on the web to provide critical infrastructure for their success. In order to facilitate these customer needs hosting companies require an EXACT method to track resources needed and consumed by every customer in realtime and historically. Currently these tools and tracking are sorely lacking.

Most web hosting companies use Unix or Linux (many varieties) for the servers that provide service for their customers. Linux is rising star and is what we use for 100% of our hosting services. Linux is GREAT! It is fast, it is stable, and it is only getting better. However, until very recently Linux has not provided the necessary kernel (heart of the operating system) enhancements to effectively track what we need to both guarantee resources for sites that are starved for cpu, disk I/O, etc, and to limit and block the cpu and I/O hogs that are causing problems for everyone else.

I talk with many hosting companies every week and try and assess what others are doing to meet these challenges. In many cases companies are just throwing more servers at a problem without dealing with the underlying cause. This was how it had to be because Linux simply didn’t provide the necessary information. Let me be specific so you know what I am talking about. Below are a list of things that we must have in order to provide guaranteed resources in a “shared” hosting environment. Some of these requirements can be obtained but are difficult to get at and others the Linux kernel either doesn’t report or give access to or is inaccurate to the point of being useless in performance tuning.

*DISK IO REQUIREMENTS**
- Byte counts per process and/or user and group to an IO device in realtime and historically (Taskstats in the kernel does this – FINALLY!!).
- Byte counts including or excluding cached hits to any physical device.
- Disk accesses per process and user per physical IO device tracked in realtime and historically – This is EXTREMELY important in a shared hosting environment. As I talk with other hosting companies this is almost never even looked at and in my view is the #1 performance bottleneck in hosting by a long shot.
- Exact stats of journaled filesystem flushing mechanisms such as kjournald, pdflush, etc.

**CPU REQUIREMENTS**
- Better hooks in the Linux kernel for managing/tracking and if necessary blocking process. “nice”ing processes is like hitting an ant with a sledgehammer. It works but it is not the elegant solution that companies need.
- Ability to reduce CPU and I/O block devices in tandem. The CSF I/O scheduler that is the current default I/O scheduler in the Linux kernel (2.6.17 and above I believe) allows I/O priority scheduling. “ionice” and other software is available to do this scheduling, but isn’t set to work with the CPU so I/O and CPU are difficult to reduce in concert. CPU and I/O don’t increase or decrease in a fixed ratio together, but viewing an individual process with a goal to reduce CPU without controlling the I/O is a waste of time in my opinion.
- Better user space application threading is also a need that all hosting companies have. As multi-core CPUs are now the norm in virtually all servers and 8 core CPUs are on the way (4 core is available now in Intel, and AMD Barcelona 4 core CPUs expected in 2nd half of 2007) more “web hosting” apps needs to be multi-threaded to make better use of these CPUs/cores. Cpanel (Our control panel) still uses many older single threaded apps because it is difficult to upgrade. The company that created Cpanel needs to take this to heart and start improving performance for the real world instead of just what works in their lab.

I am writing this blog entry for a couple of reasons. First, so you know that the CEO of Bluehost isn’t just a business/marketing entity at Bluehost. I understand the technical challenges that hosting poses. Second, I want you to know that we are actively working on and implementing changes that will shield users from other customers CPU and I/O demands. I know customers hate to see “CPU Quota Exceeded” errors. I know sites are sometimes slow or I/O is backed up. These issues are on the top of our list and solutions are coming that will be exclusive to Bluehost. We have some of the most talented and bright developers that have been challenged to solve these issues. Real solutions are coming and performance will continue to increase!

Thanks,
Matt Heaton / President Bluehost.com

I recently had a meeting with our healthcare representative for our health insurance policy at work. We talked about the skyrocketing prices, the sometimes horrific service, and the general bad state of healthcare as a whole. He then went on to say how in his opinion IHC (Our healthcare company) was so much better than everyone else in the state of Utah. I then asked him a simple question which he didn’t seem willing to answer. The question was, “They may be better than everyone else in Utah, but are they doing a good job irrespective of how they compare to others?”. He looked at me like that was the silliest question I could possibly ask.

It didn’t even occur to this person that IHC may be terrible from the customers perspective. He seemed only interested in polls and numbers that showed that IHC was better than their competitors. Who cares if IHC is better than their competitors except their board of directors. I care that MY experience was good, and up to that point my experience had been anything but good. Companies should look at themselves and see what broken processes they have and then fix them. THey must focus on their relationship with their customer and the rest will take care of itself.

I am not saying Bluehost doesn’t have areas that don’t need review. We certainly aren’t 100% effecient on support or technical areas. The one thing that we DO execute well on is looking at our company from our customers perspective and trying to fix problems that keep recurring for our user base. I simply don’t care what our competitors are doing in those areas unless they have a great idea that we can use to solve our own problems. I don’t care if our phone hold time is averaging 3 minutes and everyone else is averaging 30 minutes. I care that 3 minutes is 2.5 minutes longer than I want and I need to fix it.

The goal should be to be the best, and not just better than others, or the best of the worst. I find that many companies just don’t have the end game in sight when making decisions. We don’t always succeed and we are constantly improving, but please know that we measure our success by what our customers say not how much better we are doing than our competitors.

Thanks,
Matt Heaton / President Bluehost.com

Our organization is one of the fastest growing hosting companies in the world. We are adding about 30,000 (yes thirty thousand) domains now each month! With that type of growth we need great people to get the job done and grow with our company. Whenever we can, we try and promote people within our organization before we look outside to hire new people. Sometimes we are able to do it and other times we can’t. Sometimes we have people in the organization that I would like to promote but I can’t because they don’t take the next step in education and responsibility.

Everyone deserves a fair shot at moving up in the company, but I will never force someone. In order to progress within our company you need to show an intense desire to LEARN what is required, and then apply that knowledge to the benefit of the customer or the company within the scope of your position. Success is about DOING, not about thinking you should do it! Very few people will step up and go the extra mile when they think no one is looking. These are the people we want. This is the type of person that every company looks for (Or should look for!). This type of ability and desire can NOT be trained in my opinion. I am a very firm believer that you either have it or you don’t.

In my opinion a career is all about your value to your organization. If you don’t like your current status at a company then you make yourself invaluable to the company and ask for more money. If anyone can do your job then the job isn’t going to pay a lot and in my opinion it shouldn’t pay a lot. This isn’t an insult to the person doing a lower paying job it is a challenge to make yourself better and in the process improve your value in the job marketplace. Advancements in your career should be all about excellence in the workplace and nothing to do with age, time at the company, etc. These type of “entitlement” advancements aren’t driven by success, they are driven by a belief that if you do the same thing year after year that you should continually get raises and benefits just because you have been at a company for long time. I strenuously disagree with this type of attitude.

It should be everyones goal to make what they are worth based on skills and desire? Only YOU can and should make that happen. What are you worth?

Thanks,
Matt Heaton / President Buehost.com