MySQL Woes…

Shared hosting is a balancing act of epic proportions. Bluehost/Hostmonster/Fastdomain has thousands of CPU cores, petabytes of information (thousands of terabytes), and half a petabyte of main system memory. We try and distribute these resources as fairly as possible across hundreds of thousands of customers and close to 900,000 domains. Its a monumental task, but we have many tools to help us get up for the challenge.

We have some very good custom software that was developed in house to separate and segregate CPU allocation for users over a given period of time. Memory management is accomplished and monitored effectively by user, but there are a few areas where tools don’t exist or aren’t good enough for to do the job. Disk I/O management by user still falls short, although we are ALMOST there with respect to assigning all activity by customer. The killer for us right now and the bane of most shared hosting companies is MySQL!

MySQL is far and away the most popular database application for shared web hosts. There are many other popular and fast alternatives such as postgres, etc, but MySQL services about 95% of our customers database needs. The problem is that for whatever reason MySQL CAN NOT give us the proper detailed usage statistics we need. Breakdowns by user, and rows affected by user, and ACCURATE cpu time used by user (Not including wait times for blocked IO devices) simply doesn’t exist for MySQL. It is extremely frustrating that a database as popular as MySQL and one that strives be an an enterprise class database would lack these basic features. This one application can literally kill the overall performance of a server. So what can we do?

First, there are some patches that give some of these capabilities to the MySQL server. However, they aren’t actively maintained and they can’t be applied cleanly to the newest code bases of MySQL. As of today I have decided to have these patches picked up by paid developers and cleaned up and maintained so that they will work for current and future versions of MySQL. I will continue to pay to have these maintained and updated to serve the community as a whole. I will release the patches to the public so all can benefit from these changes (Even though they probably should be withheld as a competitive advantage for Bluehost and Hostmonster).

I will also put as much pressure as I can to have these patches added and made a part of the permanent MySQL source tree. It really is astounding to me that MySQL has not addressed these issues since day 1. If anyone has any strings they can pull at MySQL PLEASE have them consider what I have written. Turning their nose away from these problems won’t make them go away.

When these changes go live it will bring us one big step closer to having completely stable and high performing servers available for all our clients.

Thanks,
Matt Heaton / President Bluehost.com / Hostmonster.com

15 Responses to “MySQL Woes…”

  1. vinicius says:

    congratulations for the iniative in supporting opensource software! (although sometimes I think that drive-by money motivation is not as effective as drive-by heart)

    most scripts that use mysql don’t need it, they could be fairly replaced by SQLite(.org, an SQL database file) that has much less system impact in performance, memory footprint, etc and don’t require the webhosting company to setup servers.

    Having documentation with examples for SQLite in PHP and Ruby on rails could help(although I think it’s a remote possibility) Bluehost save system resources.

  2. Matt,

    Wonderful topic and thank you for being so open about such an obvious issue.

    I too, congratulate you for wanting to contribute to MySQL on a very notable issue that has indeed remained prominent throughout development, despite such focus within the industry. Your steps to help improve this technology and share it to the open source community is simply sincere and extremely commendable!

    Reading information like this only re-affirms my selection to be a part of the Bluehost empire. :) BTW, when do the new uniforms arrive?

  3. What are the difference between BlueHost, HostMonster, and FastDomain?

    Why even have three brands?

  4. Ken Fine says:

    I like and favor your viewpoints, but clients need more than 50 MySQL databases on the tag of BlueHost …. BlueHost/ Hostmonster/ Fastdomain are better than best when compare the hosting features than most of other competitors but the number of offered MySQL is less than some of the competitors, though all webmasters don’t need many MySQL databases. Please consider to offer more number of MySQL databases ….. when?

  5. Kris Beldin says:

    Matt,

    Any chance these issues will be resolved now that Sun Microsystems is at the helm? I am with a different hosting provider who just migrated servers to Colorado and thereby upgraded to a newer version of MySql and caused one of my client’s blogs to come up in reverse order!

    Talk about frustrating!

    -Kris

  6. Wes says:

    I like MySql but I think they should support stored procedure a bit more to make them more useable. At the moment my use of stored procedures in MySql is quite limited because of some of the limitations but otherwise it is a good database system with also some positives like use of standard sql.

  7. Jay R. Wren says:

    Awesome,

    Thank you for using your position as CEO of one of the leading web hosting companies to give back to the open source community on which your company runs.

    I am wondering if you have considered doing the flip side and taking the most popular web applications like wordpress, phpBB and others and hiring programmers to decouple these from MySql. It seems like an inprocess database such as SQLite might serve just as well and since it is inprocess it would allow you to measure the CPU and IO performance better.

  8. I feel a little guilty reading your post, after just last night installing RoundCube for email which required a MySQL database. I pray that it does not contribute to a higher server load.

  9. Jay Pipes says:

    Hi!

    My name is Jay Pipes. I’m the North American Community Relations Manager at MySQL (now Sun, I guess!). One of your customers pointed me to your blog entry and asked if I could do something.

    They wrote to me:

    “[Matt] seems like a sincere guy who cares about the services he provides to his customers. I don’t know if his frustration and criticism is justified…I’m a satisfied customer of Hostmonster and a happy MySQL user. I don’t want to get anyone in trouble. I hope that making you aware of this post has a productive outcome.”

    There is actually a lot I can do to help, much in the arena of making sure that your comments and criticism is heard and thought given to it. I can’t, of course, make promises about our engineering roadmap, but I can certainly help to get answers for you from the folks that control and influence such things.

    But, first, could you expand on “some patches that give some of these capabilities to the MySQL server…[that] aren’t actively maintained and they can’t be applied cleanly to the newest code bases of MySQL.”? Are these the Google patches you refer to here? Or the SHOW PROFILES patches? Could you be more specific so I can track down the specific patches and see what the status internally is? Thanks!

    Also, are you a MySQL Enterprise customer? Not that I am doing any sort of sales stuff (I don’t do sales at all..), but if you’ve got ticket history tied to your customer account, it could come in handy in identifying the issues about which you posted…

    Also, if you don’t feel like commenting on this blog post, feel free to email me at jay at mysql dot com with more details and I’ll do my best to get answers for you (and post those answers back here…)

    Cheers!

    Jay

  10. ISV Prime says:

    i’ve recently moved multiple websites to bluehost. yesterday my website was unavailable to my clients for (reportedly) 4 hrs. when i called bluehost support this morning, they acknowledged the downtime but said they didn’t know why i was down. when i asked if someone would research the issue and get back to me, the support tech (nice enough fellow) was reluctant to commit to that, but eventually did.

    are these patches related to my (thus far) unexplained downtime?

    thanks,
    hal

  11. Bob Morris says:

    This helps explains why every now and then I call tech support and tell them that my MySQL is down. It’s because you don’t have the right tools to track it real time, apparently because they don’t exist. Yikes.

  12. David Mackey says:

    Glad to hear you will be making this code available as open source.

  13. I have friends at MySQL and after I read your post here, I wrote them explaining the needs for this issue. I hope that some time soon they will decide to incorporate such changes to the main product code base.

  14. falsedragon says:

    So this is why I’m having problems lately? I don’t get MySQL errors usually though, the domain becomes completely inaccessible.

    I’ve been with Bluehost for almost a year and at first it was great, loved it completely. For the last few months though I’m noticing a lot of downtime and whenever I contact live support about it they just say “we’re looking into it, it should be fixed in a few minutes”. Other people I know have no problems what-so-ever so I guess I’m just stuck on a crappy server or something.

  15. meizitang says:

    I’ve been with Bluehost for almost a year and at first it was great, loved it completely. For the last few months though I’m noticing a lot of downtime and whenever I contact live support about it they just say “we’re looking into it, it should be fixed in a few minutes”. Other people I know have no problems what-so-ever so I guess I’m just stuck on a crappy server or something.

Leave a Reply