mattheaton.com

The affiliate market is a very important part of Bluehost’s business.  Affiliate marketing was our chosen method to grow Bluehost.  Of course we did direct marketing and we hoped that our word of mouth business would continue to grow.  All in all it has worked out really well.

There were two reasons our affiliate marketing worked so well.  First, I think our hosting package was/is superior to our competitors.  This made it easy to promote our services compared to our competitors.  The other important reason that we did well was because our tracking was more accurate than our competitors.  We made sure there was no way that a sale would not be counted.  We wanted our affiliates to make a ton of money because we knew that would ensure that our business continued to grow.

Over the past 3-6 months we have noticed a very interesting change in behavior.  Private browsing, anonymous surfing, etc is WAY up.  Not  only is a significant amount of traffic not using cookies (At least not cookies that live more than 30-60 minutes), but we are seeing a greatly increased number of users ip addresses that are changing multiple times during a single page loading.  This has always happened in the past, but it was an insignificant amount of traffic.  That just isn’t the case anymore.

The majority of tracking has always been done by cookies, and then if they are not present to fall back to IP address.  That is no longer good enough.  Effective today, we will now be tracking using the traditional cookie, and ip tracking, but we will also use a method that can punch through virtually every anonymous or traditional proxy server, or series of servers, and give us real identifying info.   This has yielded a fairly substantial increase in tracking accuracy.  I have debated myself all morning on whether I would give the technicals details of how this works, but I have decided against it.  I think this is a competitive advantage for us.

This means that effective immediately many affiliates will see a measurable increase in conversion rates from all our hosting brands including Bluehost, Hostmonster, and Fastdomain.  Thanks to all of our more than 50,000 affiliates!  You are the reason we have been able to grow so quickly!

Matt Heaton / President Bluehost.com

Congratulations to us!  Our organization is now hosting more than 1.5 million domain names and nearly 2% of all domain names in use on the whole internet.  I think that is amazing.  Thanks to everyone that uses and recommends our services.  Hope to post when we reach 2 million real soon!

Thanks,
Matt Heaton

Seems I am the devil because I didn’t put some code up that I said I would.  So here it is.  This is a patch to the 2.6.27 kernel that outputs all the files that have been modified, chmod, created, etc.  Basically, all the files that would need to be backed up WITHOUT having to do a full system scan of a block device(Hard drive, etc).  You can then use that list to know what files to back up.  The backing up of the files is not part of this patch.  That portion is the easy part.  Exporting the files from the kernel was the harder part, and that is what this patch does.
There is also no hashing, sorting, or filtering of any kind so duplicate files could be reported - meaning a file could be modified and then before you backed it up become modified again.  In that case it would output the file twice.  Simply, remove the double entries to solve this issue.

If you have issues with this patch please let me know, but please don’t ask for feature enhancements or code changes unless it is critical to the use of the patch.  We use a modified and updated version of this patch in house with a little different functionality and don’t have a lot of resources to keep updating a patch that we no longer use.  Also, please don’t take that to mean this patch doesn’t work well, because it is fantastic for backup purposes.   I only mention that we use a different version because of other functionality that we needed for a different purpose that we have combined with this patch.

Here is the link to this patch -

http://www.mattheaton.com/backup-patch.rtf

There are now 2 on/off switches:
/proc/sys/fs/bh_logging_user and
/proc/sys/fs/bh_logging_root

They are off by default so there should be no affect on the systems until they are turned on.

To turn them on simply echo 1 to the proper file as shown below -

echo 1 > /proc/sys/fs/bh_logging_user
echo 1 > /proc/sys/fs/bh_logging_root

It also requires a change to /etc/syslog.conf to direct the messages somewhere.

Here’s the addition I made on bhtest:
kern.debug    /var/log/modified

Thanks,
Matt Heaton / President Bluehost.com

Hosting is an interesting business.  It has come a long way from the free website days of Geocities and the like.  Speed, reliability, and features have grow far beyond what they were only a few short years ago.  What you can now buy for under $10 a month is 10x as powerful as what you could buy 5 years ago for the same money.  That is a good thing.

Hosting also has some short comings that still haven’t been addressed that hamper many individuals and companies with lower priced shared hosting.  I would like to discuss some of these short comings and how I believe they can be fixed and show how we think we have finally been able to achieve shared hosting nirvana

Listed below are the well known and almost impossible problems to solve with regards to shared hosting -

1) Resource Allocation - Because many users are on a single system it is hard to allocate resources properly.  You may have a very underutilized server (Resources going to waste) or an overcrowded server that can become bogged down with overuse.  It is impossible to determine the usage of a particular user in advance.  This causes unreliability on a server because a server can spike out of control at any time.  Common problems are too much disk i/o consumed in bursts by particular users, extreme short or long term memory usage by a particular user, or spiking or prolonged cpu usage by a small group of users on a particular server.

** What we can do about it -

To us it all comes down to instantaneous real time tracking (About every 15-30 milliseconds in our case).  Advances in general linux kernel tracking and our own propriety tracking have finally allowed us to know which users need the most resources.  We have then tried to automate as much of the process of allocating free resources in real time to these users as possible.  We are constantly updating these automated tools to make the process as seamless as possible.

2) Tracking User Consumed Resources - You can’t “blame” users or even monitor many user activities (Cpu use, or disk i/o usage) on shared servers because tracking user resources means you have to know what user ran what process.  The problem comes into play when you have major applications that don’t run as a particular user and instead are run as their own process with no “real” user tracking.  Let me give you an example of the two biggest problem applications in this area - MySQL, and Apache.  MySQL consumes an enormous amount of CPU and Disk I/O resources, but it normally runs as the “mysql” user.  This makes it very difficult to track resources used by a particular user.  Built in tools such as slow query logs, etc are extremely inaccurate in measuring disk i/o and CPU usage by user.  The other major culprit is Apache.  Apache runs as a separate user as well (At least in our case, and most web hosts have a similar setup).  Apache spawned script process such as PHP, Perl, etc can easily be made to run as a specific user for tracking purposes, but the Apache processes themselves and the corresponding cpu and I/O overhead is never attributed to a single user.  There are many applications that fall into this category, and all of them make tracking inaccurate and problematic for hosting companies.

** What we can do about it -

I have spent thousands of hours over the last 3 years working personally on this problem and I am VERY happy to report that we have nearly solved this problem (MANY thanks to kernel developers all over the world that helped out as well as the talented developers in house!!).  We have spent considerable time and money modifying the linux kernel and userspace applications (MySQL, Apache, etc) to report exactly which user is responsible for cpu and i/o usage in real time.  Lets give an example - Lets say we have user “matt” that does a MySQL query that take 2 minutes to complete (clock time) and 90 seconds of real CPU time to complete (Actually number of CPU seconds required to complete the query).  When MyQL passes the query to specific thread to be serviced we start tracking for that particular user the EXACT cpu time that was used, and the exact number of system reads and writes as well as device specific reads and writes.  We can use this to track and slow down the extremely heavy users in real time so that the server is calm and available for everyone to use.  We are in live testing right now on several boxes and hope to have the CPU portion of this rolled everywhere in the next 3-4 weeks.  The disk I/O portion of our code is already live on 90% of our system and will be live on 100% of our servers in the next week.  The importance of this can’t be overstated.  This is what shared hosting has needed forever, and what will allow it to compete with and in many cases top VPS in performance while maintaining stability in the system.

3) Immediate action on policy enforcement - This is a BIG deal with shared hosting!  When a specific user “violates” a policy like excessive CPU usage or disproportionate disk I/O or memory usage 99.9% of all shared hosting companies will try and alleviate the problem by killing processes or banning a user AFTER the damage has already been done.  It does no good to ban someone after they have consumed so much CPU that the server becomes sluggish.  The sluggishness or downtime has already happened at that point.  Most hosting companies have a very difficult time every determining where these cpu and I/O abuses are coming from let alone mitigating the problem before it happens.  Virtually NO SHARED HOSTING COMPANIES  have good options to actively slow down cpu usage - They usually just stop the offending processes (Not a good option), or kill processes consuming too much disk I/O - not a good option either.

** What we can do about it -

As mentioned above in item #2 we can now track and monitor cpu and disk I/O usage in real time for all our users.  Based on this information we can now do what no other shared hosting company has ever been able to do effectively.  We can limit disk I/O activity in real time and limit CPU activity in real time for all our users.  This allows us to mitigate the effects of sudden spikes in usage that would normally affect all other users.  Here is a good example to illustrate this point - Lets say we have 100 users on a server and that 99% of the time everything runs smoothly but one day one of the users makes it to the front page of digg.com.  One of two things are going to happen.  Either the user that is causing the excessive load is going to be shut down or the server is going to be sluggish or possibly down until traffic subsides to that site.  We have been in this position many times in the past and its never good for anyone.  The disk I/O portion of that problem is now solved for us, and the CPU issue should be ready in the next few weeks.  Instead of shutting the user down we can now “contain” them as if they were on a VPS.  We can isolate them from other users  so they don’t cause problems but still allow them to use any extra CPU cycles or I/O operations that are available.

**What does this mean?

What does this really mean?  It means we will very soon be able to offer the VPS experience for less money and less hassle than every other VPS product out there, and that our shared hosting product is about to become a LOT more stable than everybody else out there.  My opinion is that most VPS users really don’t need or want root access that requires their own time for security updates, Cpanel updates, etc.  They simply want a contained environment and guaranteed resource allocation.  We will be able to offer guaranteed resources just like a VPS or dedicated server solution without requiring any changes for our users.  I am very excited to see all this materialize as this has been my pet project for several years.  If you have made it this far in the blog entry I congratulate you for your tenaciousness in plowing through my technical ramblings!

Thanks,
Matt Heaton / President Bluehost.com - Hostmonster.com

I guess I just don’t have much self control.  It seems I can’t go six months without writing something negative about Microsoft.  I don’t think its my fault.  Microsoft just makes it so easy sometimes.  Lets examine a couple of quotes by Microsoft management and then you tell me if I’m wrong or not (Have I ever been wrong )

Here is the man himself - Steve Ballmer

“Apple gained about one point [in global market share], but now I think the tide has really turned back the other direction.  The economy is helpful. Paying an extra $500 for a computer in this environment — same piece of hardware — paying $500 more to get a logo on it? I think that’s a more challenging proposition for the average person than it used to be.”

This is the CEO of Microsoft?  Here is my beef with this statement.  First, while Apple does demand a premium for their products $500 is a modest exaggeration given the quality of the parts Apple uses, but my real issue is that he seemingly doesn’t understand at all the value that Apple brings to the table.  Ballmer very unwisely compares the hardware that comes on a standard PC with Mac hardware and then assumes that the operating systems are of equal value.  In short he says a PC that comes with Vista for $1000 is a better value than a Mac with the same hardware that comes with OSX that costs between $1000-$1500.  The problem is that almost no Mac users would agree with that statement.  I’m sure I am a little out of the norm, but I would happily pay 5x the price for a Mac because I have no patience for Microsoft’s products and their constant problems.  If he REALLY believes that a few hundred dollars are going to sway a significant portion of Apple’s user base back to Windows he is grossly mistaken.  People are not only buying Macs, but they are increasing their share at TWICE the rate of HP and Dell.  Why?  Isn’t this a recession?  The reason is because they recognize VALUE.  Notice I didn’t say cheaper.  Apple’s operating system is so much better than anything coming out of Redmond these days that people are switching at a record pace.  Microsoft has put themselves in the poor position of having to compete on price because they can’t compete on quality.  Hopefully Windows 7 will improve things and force both companies to improve their products at a rapid pace.

Ok next quote - IE General Manager Dean Hachamovitch

“We’re going to keep making the script engines faster (but) right now it’s not clear how many people are gated by script performance.  JavaScript comprises a small portion of how fast a Web page will render. It is a piece, but by no means the holy grail.”

I don’t think this guy could be more wrong.  IT IS THE HOLY GRAIL.  He is SO wrong that I’m willing to put $10,000 on it.  If Microsoft doesn’t publicly say in the next 12 months that they are actively working on making javascript faster and that javascript is the primary driving factor for web based services and that fast javascript execution is the key to making web based services usable then I will donate $10,000 to the Make A Wish foundation in Microsoft’s name.

The reason web based applications are limited in scope and use is DIRECTLY related to javascript speed.  The issue is no longer bandwidth or processing power, it is the javascript engine.  There is a reason why Mozilla spends a great deal of time on TraceMonkey, and Apple on Nitro (Webkit), and Google on Chrome’s V8.  They know that the javascript engine is what will drive the use of their browsers in the future.

Microsoft is more than likely simply downplaying this because their javascript engine is weak compared to their competitors.  They better secretly be taking javascript performance very seriously now or they will soon find themselves too far behind with an inferior product (Again…) because of their lack or forethought.

Matt Heaton / Bluehost.com

I have written over and over that great things are born from trials and failure.  I have also written that very often when you need something done right you just do it yourself!  Although web hosting backups don’t count as one of life trials they have been a pretty big thorn in my side.

As most of you know we use Cpanel as the backend system for our control panel.  Is works reasonably well for most functions, but one area where it falls flat on its face is backups.  We have been using our own system for a long time and the performance is adequate but far from what we thought was ideal.  We tried several open source tools as well as a paid option from R1Soft (The WORST software I have ever had the great misfortune of trying).  Nothing was fast enough and here’s why.

Our average /home partition on one of our servers has anywhere between 3-6 million files on it.  Lets assume that only 10,000 files on that partition were modified in a 24 hour period.  You still have to parse/scan the directories that contain those 3-6 million files just to find the 10,000 files that were modified.  Without copying the files and doing just the stat (Or the scan) of all those files takes hours to complete.  However, what is FAR worse are the seeks that the block device (Hard drives) incur while they are doing the scan.  It slows the system way down just to find the files to backup.  Then when it finds files to change it still has to copy them.  This is “just the way it is” on every system I know including solaris, windows, linux.

One day I was thinking (In the shower of course - since 90% of all good ideas come to you in the shower) why not just have the linux kernel dump the name of any file that was created/modified (Any bytes written) at the time the file was modified and use that as a list of files to back up.  The kernel already has this information when a file is updated and just throws it away.  There is virtually no overhead to do this and it saves literally 90% of the time we would normally have spent on the system doing backups.  There are already similar hooks in the kernel to get this data through innovative techniques like Inotify, but Inotify is capable of a lot more than what we wanted and consequently MUCH slower.

The problem was that I couldn’t find a single program that implemented this idea or even mentioned this technique anywhere on the web.  In situations like this I turn to our favorite in house kernel hacker and demand magic.  In this case it took him one day to write a kernel patch that implemented this and stripped out files files that didn’t matter for backup purposes such as /proc or /dev and so forth.  So does it work?

YES!  Whats really neat is that it isn’t anything that is specific for Cpanel.  It works for any linux filesystem such as XFS, EXT3, EXT4, Reiser, JFS, etc.  I think that most admins don’t fully realize the amount of time that is wasted and I/O that is consumed just in the determination of what files need to be copied.  This new backup method is literally 10x faster than what we had before and puts far less load on the server in the process.

So what to do with it.  Well, after we clean up the kernel code a bit and make sure it is 100% rock solid I will post the patch free of charge here on my site.  The patch simply dumps a list of files to be backed up to any file you specify.  You can then do whatever you want from that point.  We will have a fully implemented Cpanel backup that will work perfectly with Cpanel and is completely compatible with their restore feature.  I have no pricing for it, but I will tell you this.  I will charge you only 25% of the lowest price that you are quoted from R1soft for their horrible software.  Meaning if you are paying $50 a client license I will charge you $12.50 .  Of course you are more than welcome to use the patch free of charge and implement your own system.  It is the fastest solution of any system I have ever tested (Including of course R1soft).

If you have any ideas that you think could make the product even faster I am open to any of your ideas.

Thanks,
Matt Heaton / President Bluehost.com

I can’t take it anymore!  If I read/watch/listen to another news program that talks about how our government (Read: Me and any other tax payer) here in the United States should bail out yet another industry I’ll go nuts.  I guess I am just so far out of touch that the ideas that seem fundamental and core to me are now “out of date”.

How dare I believe that both government and the private sector should be responsible for their actions.  How dare I believe that consequences for your actions should always be accepted.  How dare I believe that people should be allowed to succeed or fail based on the merit of their ideas and their ability to effectively implement those ideas.  As I said before - These ideas are now “out of date” and are almost an insult to half the population in the United States.

To me it seems that we are weak and becoming weaker all the time.  The bar for what is acceptable is constantly being lowered to accommodate those that can’t meet expectations.  The consequences for our actions are constantly mitigated so that people won’t “suffer”.

If I tell one of my sons that the pot is hot and not to touch it I hope he will learn from what I say.  If he doesn’t listen he burns his hand and in the process learns two lessons.  He learns that I care for him enough to tell him how to avoid being hurt, and I guarantee that he finally learns that the pot is hot.  Do I want him to burn his hand?  Of course not.  Bad decisions can/should be painful to endure, but not taking your licks when you make that bad decision is worse.  You learn to be weak and you avoid the natural consequences that act as the teacher of life’s valuable lessons.  Our society seeks to dodge consequences at every turn and in so doing rewards failure at the expense of the successful.

I feel very strongly that our government and many of our people look for a way out of their problems that bypass responsibility completely.  Not every business should be saved.  Not every need of the people can and should be paid for by government.  Not every painful experience should be avoided.  We learn because of mistakes that we make and change because of what we learn.  When we are constantly told that what we are doing isn’t wrong and that it isn’t our fault that we are in the position that we find ourselves in then we can’t learn and move forward.

As far as I’m concerned we are headed backwards in a dramatic way, but what do I know?

Matt Heaton / President Bluehost.com