R1Soft is a backup “solution” aimed at companies that want to backup only the changes that have been made since the last time a backup was created. Its a good idea, but it went horribly wrong. I am writing about it today because it was such a poor experience working with them and we had so many server problems and REPEATED DATA LOSS that I felt compelled to warn other hosting companies and customers about this extremely poor product.
R1soft caused enormous I/O load that literally killed our servers off and on multiple occasions simply would not restore our data and caused 100% data loss on the servers we had “backed up”.
The rest of this blog entry details technical reasons why R1soft is broken and inept. If you aren’t interested in those details you can skip the rest of this entry.
MAJOR TECHNICAL DEFICIENCIES AND OPERATIONAL ISSUES WITH R1SOFT –
1) Buagent (Their software that records what has changed) is a HUGE i/o pig. It constantly blocks on I/O when there are changes causing every other drive or array in our system to lag while it catches up. R1soft told me it “works great” and to send them over details and they would “Look into it”.
2) Buserver (Their software that writes what has changed) is INCREDIBLY slow in every way that it is virtually unusable when you have a lot of data to back up with tens of thousands block changes every day per server. Clearly it hasn’t been tested well under sustained and real heavy load – at least in my opinion. There are multiple kernel and userspace methods to speed this up, but again it falls flat on its face when trying to keep up with our load.
3) Restores are buggy and VERY unreliable. We have attempted more than 20 restores (whole partitions) and had more than 30% be a complete failure, meaning TOTAL data loss. Doing user level restores were the same story – incomplete backups, unreliable and SLOW restores.
4) R1soft support is lackluster to say the least. When I complained they said, “You only opened 17 tickets in the last 60 days.” 17 tickets? Often when we had issues doing a restore or other problems we would call and get no one to answer the phone. Its comforting to know that when you can’t do a restore for 500 clients that we can possibly talk to someone on Monday who will blow us off and tell us that the newest version of the software MIGHT fix our issue. In the meantime they apologized for losing all our data over and over again. Wonderful.
5) The software is based on filesystem block changes. The problem is that the kernel and ext3 and reiser are changing all the time. As we made updates it constantly broke R1soft. When they would finally fix the problem it usually required us to reseed the entire backup again losing everything that we had previously backed up and putting enormous load on the server again while it started over from scratch
In all my years of hosting I have never had such a bad experience with a “partner” company as R1soft. I apologize to my customers for subjecting them to this software abomination and to my staff for making them deal with it for 2 months before we finally extracted the final traces of this software from our system.
Matt Heaton / President Bluehost.com / Hostmonster.com