Archive for July, 2008

R1Soft – A Backup Nightmare

Thursday, July 31st, 2008

R1Soft is a backup “solution” aimed at companies that want to backup only the changes that have been made since the last time a backup was created. Its a good idea, but it went horribly wrong. I am writing about it today because it was such a poor experience working with them and we had so many server problems and REPEATED DATA LOSS that I felt compelled to warn other hosting companies and customers about this extremely poor product.

R1soft caused enormous I/O load that literally killed our servers off and on multiple occasions simply would not restore our data and caused 100% data loss on the servers we had “backed up”.

The rest of this blog entry details technical reasons why R1soft is broken and inept. If you aren’t interested in those details you can skip the rest of this entry.

MAJOR TECHNICAL DEFICIENCIES AND OPERATIONAL ISSUES WITH R1SOFT -

1) Buagent (Their software that records what has changed) is a HUGE i/o pig. It constantly blocks on I/O when there are changes causing every other drive or array in our system to lag while it catches up. R1soft told me it “works great” and to send them over details and they would “Look into it”.
2) Buserver (Their software that writes what has changed) is INCREDIBLY slow in every way that it is virtually unusable when you have a lot of data to back up with tens of thousands block changes every day per server. Clearly it hasn’t been tested well under sustained and real heavy load – at least in my opinion. There are multiple kernel and userspace methods to speed this up, but again it falls flat on its face when trying to keep up with our load.
3) Restores are buggy and VERY unreliable. We have attempted more than 20 restores (whole partitions) and had more than 30% be a complete failure, meaning TOTAL data loss. Doing user level restores were the same story – incomplete backups, unreliable and SLOW restores.
4) R1soft support is lackluster to say the least. When I complained they said, “You only opened 17 tickets in the last 60 days.” 17 tickets? Often when we had issues doing a restore or other problems we would call and get no one to answer the phone. Its comforting to know that when you can’t do a restore for 500 clients that we can possibly talk to someone on Monday who will blow us off and tell us that the newest version of the software MIGHT fix our issue. In the meantime they apologized for losing all our data over and over again. Wonderful.
5) The software is based on filesystem block changes. The problem is that the kernel and ext3 and reiser are changing all the time. As we made updates it constantly broke R1soft. When they would finally fix the problem it usually required us to reseed the entire backup again losing everything that we had previously backed up and putting enormous load on the server again while it started over from scratch

In all my years of hosting I have never had such a bad experience with a “partner” company as R1soft. I apologize to my customers for subjecting them to this software abomination and to my staff for making them deal with it for 2 months before we finally extracted the final traces of this software from our system.

Matt Heaton / President Bluehost.com / Hostmonster.com

My second wind…

Friday, July 25th, 2008

I have been doing hosting for almost 10 years now. I can’t believe it. It really only seems like a few years when I think about it. During that time I have started three hosting companies (Actually five if you count hostmonster and fastdomain as separate businesses which they are) and sold two of them (50megs.com, and 0catch.com).

Bluehost and Hostmonster are my primary focus, but I do have other businesses that take some of my time (SimpleScripts.com) as well as real estate interests. The past couple of years have been interesting for me. Health concerns have held me back more than I care to admit and often times I have “sprinted” for a couple of weeks and then took time off to get ready to go again. I hope to put those issues behind me soon.

For the first time in a while I really feel ready to get out there and start crushing the competition again. Anyone who knows me knows I’m at my best when I’m obsessed with winning. I’m ready to start obsessing again :) I’ve been scheming and plotting knew ways to get a LOT more signups each day. I’m very excited to work on kernel changes and improve technical advances in shared hosting. In addition, I just purchased a 50,000 sq ft buliding to move our business into and will begin construction on our own data center in a few months. When the new data center is in place we might actually get around to doing VPS hosting as well – If we do, watch out! Our VPS method of doing things is different from what others envision and will be VERY good.

I guess I finally got my “hosting second wind”. Its about time! I thought I was getting too bored with hosting, but it turns out I’m just getting started.

Matt Heaton / Bluehost.com

Iphone “security”

Friday, July 25th, 2008

The Iphone brings a host of new features with this release, but also brings some interesting security issues with it. Some are new while others have been around for a awhile. Sometimes I like to poke around with a digital stick and see what I can find…

Some people know that I have a somewhat “checkered” past when it comes to hacking. FBI/Law enforcement confiscating my computer aside, (Hey, I was a minor at the time!) hacking can be fun. In fact, I love it! Ok, down to business. The new Iphone “GPS” is actually an a-gps. This is good. A traditional GPS can be slow to locate you and so the Iphone can first try and triangulate your position using traditional GSM cellular antennas. In addition Apple has partnered with Skyhook (http://www.skyhookwireless.com) to track wireless APs (Access point – your linksys wireless routers etc that you have at home or work). Skyhook basically takes these APs and tags them geographically so they can report approximate positions to Apple. This means that its very likely the APs/Routers that you use at home and work are in Skyhooks database.

The problem is with the “Locate me” feature in google maps. Your phone queries Skyhooks database for the info. Every phone, actually every ethernet network device has a MAC address that is “supposed” to be unique. The problem is that MAC addresses can be cloned. I wondered if there was a way to “trick” Skyhook using the MAC address on a phone to give me the current location of an Iphone that I didn’t have. Being the lazy hacker that I am I googled for info on my “project”.

LONG STORY short, you can fairly easily set up a bridged network that is “new” to Skyhook and then “ask” Skyhook to tell you where you are. Several others were WAY ahead of me on this idea and had even written scripts to help with the process.

Now I have no real reason to care one way or the other about the Iphone “Locate me” security. I write this blog entry simply because I find it fascinating that its possible to track(Stalk) almost anyone you want these days. Real digital security is a myth. At best you can make it too “painful” in time or resources for someone to want to hack your system. Its always been a cat and mouse game. If one mouse gets caught there are always 100 to takes its place :)

Matt Heaton / President Bluehost.com

Microsoft… Yes again…

Saturday, July 12th, 2008

Its been a while since I bashed Microsoft, mostly because I don’t pay attention anymore. However, every once in a while a statement comes from Microsoft that is so ridiculous in my opinion that I just can’t resist. Its almost like they write this stuff just to tempt me to mock them -

Here is a nice quote by Kevin Turner the COO at Microsoft -

“We’ve talked a lot about compatibility, we don’t need to talk as much about compatibility anymore, we need to talk about the fact that, look, what Vista is, its the most secure product in the history of operating systems on a desktop. It is more secure today than Apple Leopard, or XP, or Linux, or open source. We built this product to engineer in security on the front end, not as a service pack. As a result of that, we tightened down things like user account controls. Yes, it required a lot of compatibility upgrades and fixes, but you know what, it’s important that you understand the progress, and you’re able to articulate that, and fewer patching is what all customers want, and there’s a cost savings there. Windows Vista with Service Pack 1 delivers that.”

Now I put this out to you – Is there ANYONE on the planet that believes that Vista is MORE secure than OSX or various flavors of linux based operating systems? Anyone…. Anyone… In addition, last time I checked “linux” was a kernel not an operating system, and “open source” is a concept for software freedom and distribution not an operating system. I’ll just remember that Microsoft “told me” that Vista is more secure the next time I have to fix one of my kids computers that got turned into a zombie AGAIN just for visiting some website.

He also talks about compatibility as if they’ve solved the issue? Does he even use Vista? Vista should either break compatibility in the name of stability and security OR it should maintain compatibility at the expense of stability and security. Instead its backwards compatibility isn’t great and neither is its security, and ESPECIALLY not its stability.

This is Microsofts COO talking?! When you have have a tree this big and this rotten you don’t prune anymore, you simply chop it down and start over. Microsoft could simply admit where they have issues and work to fix them. Customers don’t like that, but at least they UNDERSTAND it. Instead Microsoft will scream at you over and over that the sky is green not blue. Spinning, PR alerts, and bald faced lies just don’t work anymore.

Ahh… I feel much better now.

Matt Heaton / Bluehost.com

So close I can taste it…

Saturday, July 12th, 2008

Shared web hosting is a changing animal. In the mid 90′s it was nothing more than some server space with a few tools to manage your files and FTP access. Today shared hosting can be as simple as a placeholder page for a newly purchased domain name to advanced back end systems with multiple databases and encrypted voip traffic. One thing that hasn’t changed is that 1% of the users cause 95% of the server issues and 95% of the headaches that all users experience in relation to server performance and downtime.

This is going to change very soon and I couldn’t be more excited!! For years I have been writing about kernel issues with linux when it comes to disk I/O tracking/shaping and monitoring. I have complained about MySQL’s inability to monitor REAL CPU and REAL I/O by user and individual databases historically or in realtime. I have vented that the linux kernel had no real metrics to break down cpu and I/O by individual user or by process over a historical period or in realtime.

Up until now we have dealt with each of these issues in a variety of ways. Some of our solutions were elegant while others were clumsy and somewhat inaccurate. Times are changing. For the first time ever I feel like I have REAL solutions to every one of the problems mentioned above. More importantly I can track and find these issues in realtime as they are happening without putting any load on the server to glean the information.

This might not seem so important to those of you reading this entry, but in my opinion it is the single greatest thing to happen to shared hosting since its inception. Thats a bold statement. Shared hosting by definition has been shaky and unstable at times for the simple reason that so many people are using the same resources at the same time. For nearly every hosting company the strategy has been to track what you can, but fix most of the problems caused by users minutes or hours after the problem has already occurred.

It doesn’t have to be this way anymore. I know because we have many tools that find these problems within a few seconds. In fact, we have the tools to find everyone of our common ailments in less than 5 seconds. I just have to finish having the tools written to act on this information to relieve server problems in realtime. That is the easy part – that is why I am so excited. I have personally spent thousands of hours researching, testing, coding, testing, researching, testing, and testing some more to solve many of these issues. It can be done.

Most of these problems can be attributed to less than 1% of our users (Actually about .5%). Most of the problems aren’t intentional on the part of our users but are created by them nonetheless. We are light years ahead of the competition in shared hosting when it comes to tracking and solving these issues. In a few months time (Most in less than 30 days) as these changes are rolled out across our servers you will notice SUBSTANTIAL increases in both uptime and server speed.

We have invested a huge amount of money and time with in house developers, kernel developers, and custom tools to address these issues. So far many of these changes have been good, but soon you will see what we are capable of and what drives us to be better. Other shared hosts won’t be able to compete with the changes we are putting in place. Blood, sweat, and tears and a lot of sleepness nights can’t be duplicated by other hosts in a few months time. Thanks for sticking with us, and be on the lookout for MUCH better server performance!

Matt Heaton / President Bluehost.com

I’ve been asked to give so little…

Friday, July 4th, 2008

Ive been asked to give so little…

On this 4th of July I look back at what has been sacrificed by so many others for the freedoms and benefits that we enjoy today. I am someone that is always looking at how to improve something, how to make something better. In doing that, I often forget what was required to get us to where we are today.

Not today, today I express my absolute sincere appreciation for our armed forces that have in many cases given up their lives so that I might by able to pursue my life and raise a family. When I look at my life in comparison to these brave men and women and I am embarrassed at the opportunities that have been offered to me at the expense of our soldiers.

I am eternally grateful. I love you for your dedication to your country and freedom, and I promise that if the same requirements were put upon me that I would follow the same course to uphold the values we hold dear and to ensure the livelihood of the freedom we all enjoy now.

Freedom is a gift from God, but must be protected by man. God bless every one of our servicemen for your service on our behalf. I thank you, my family thanks you, and all Americans thank you for your service!

Matt Heaton / Bluehost.com