I've been nothing but impressed with the service I've got from WebFaction and the reliability I've gotten from their servers. I even had a very high traffic site run on a WebFaction shared account without a hitch.

Today, however, we got a first hand look at the downside of a shared server. If you're a webfaction customer you should be (no really, you should) subscribed to their status blog It's a great way to be kept up to date on any issues that might affect your site. The most recent issue has to do with the MySQL server on web49: the server that we happen to be using to develop a very large project. As you can see from reading the entry, the problem appears to have been caused by a corrupted database table (not one of ours) which was causing some unreliability with the database server (our Django based site was intermittently unable to connect to the database and, when it could, intermittently unable to authenticate). Though they thought they had it fixed, the problem returned and while they're attempting to fix it for good they've rolled back the entire database server to a known good backup.

This is a good approach as it means that everyone should still have most of their data in the meantime. Unfortunately, we happen to be in the middle of populating the database with the data we need to go live in the near future. Rolling back the database even by a day means that we've lost a ton of work. We should get it all back once the problem is fixed, but of coruse that means that we have to put the work on hiatus until the problem is fixed to avoid versioning issues.

This right here is the perfect illustration of why a dedicated server is a good idea. Yes, a shared server might be able to support your site. But it also leaves you vulnerable to the actions of the people you're sharing the server with. If someone else does something stupid that corrupts a database table on a server that they share with you suddenly you stand to lose a lot of work. If someone has a poorly written app that somehow manages to crash the server or even just eat up all the RAM, your site goes down. With a shared server you simply don't have the security of knowing that your site is stable and secure even if you trust your hosting company and you trust your code.

That security is what you're paying for when you get a dedicated server over a shared one.

I've been using WebFaction for my hosting for a while now, and have been extremely pleased with them. In addition to the fantastic service I've received, I've been very impressed with the intelligent way they have their servers setup. They've clearly done a lot of work to make things as modular as possible which makes it insanely easy for me to run multiple sites with very different requirements seamlessly on the same server.

Basically what they've done is segment out all of your different websites into 'applications'. Each application in represented as a directory in your ~/webapps/ directory, and is essentially a self-contained environment with it's own apache instance, and, in the case of a Django app, it's own $PYTHONPATH. The end result is that even though all the websites are being stored and run from within my home directory, they're entirely modular, can have different, or different versions of the same, dependencies installed, and can be shut down and restarted independently of one another. On top of all this is a fantastically simple custom web-based control panel that I'm pretty sure is built with Django.

I've been so impressed with how well this setup works, that I've decided to duplicate it on my home server for development purposes. Currently I do pretty much all my development work on my Gentoo Linux powered ThinkPad. To that end I've installed Apache, MySQL, PostgreSQL, SQLite, Python, PHP, &c.; to allow me to mimic the live sites as closely as possible and to allow me to continue working when I don't have internet access (such as when I'm flying or visiting Jessi's family out beyond the reach of broadband). This works very well, but as I'm just using a basic Apache install, without any VirtualHosts, it's not nearly as flexible and means I can really only work on a single site at a time with some work necessary to switch back and forth between projects. Of course most of the time I just use Django's built-in development server when working with Django, but I do end up relying on Apache sometimes, and I'd like to set up my home server as a more complete development environment for both myself and some friends I can grant VPN access to. So to that end I've been looking into WebFaction's setup with the idea of re-implementing it myself.

Turns out it's pretty simple. Simple enough that I almost feel like I should have thought of it myself. Basically, WebFaction's setup scripts create a new 'app' in your ~/webapps/ directory, and populate it, most importantly with a copy, owned by your user, of the Apache executable, some scripts to start, stop, and restart that executable, and an httpd.conf file that sets the (in the case of a Python-based app) $PYTHONPATH variable to include a ~/webapps/yourappname/lib/python2.5/ directory allowing each site to maintain it's own dependencies independently (you can also put things in your ~/lib/python2.5/ for global dependencies if you want). Oh, each application also gets it's own copies of the necessary Apache modules to the same effect. Each application's Apache instance(s) is set up to listen to a different (non-80) port. The end result of this is an extremely simple, extremely modular setup that works fantastically.

Obviously I've left out a step here. If each Apache isntance is listening to a different, non-80, port, how does your traffic get to your actual site? This is the one part that I can't really just peek into the configuration files for, because it doesn't (as far as I can tell, which makes sense) live on the same server as my sites. I assume that what's happening is that WebFaction's name servers are simply pointing requests to (for example) joshourisman.com:80 at my.webfaction.server:portnumber. Again, a simple, yet elegant solution that allows for easy customization and expansion.

I haven't yet tried to implement this setup myself (I first want to move my server from FreeBSD to Linux (which now that I'm using full-time again I'm just much more familiar with), but there's nothing about it that's particularly tricky. Really, the routing is probably going to be the hardest part, but I'm planning on replacing our rather lackluster TrendNet wireless router with a Linux box which will give me much greater control and (hopefully) better reliability.

As I mentioned in my last post, I recently migraded my dy/dx tech website to a different hosting company. If you've really been paying attention, you may recall that not too long ago I had gotten a Media Temple hosting account with the plans on migrating all of the sites I host, both my own and clients' to it only to discover that setting up Django on a Media Temple (dv) account is far more trouble than it's worth. My estimation of that hasn't changed, in fact I actually cancelled my Media Temple account a few weeks ago after the last client I had hosted there was moved off to another host. My experiences with WebFaction have been so positive (exploding data centers notwithstanding), that I have instead migrated everything to their servers. Well, not everything yet. This blog is still hosted on DreamHost for the time being (though I plan on moving it to a WebFaction hosted WordPress blog in the very near future before eventually migrating it to a Django based solution as I've mentioned before).

The hosting hassles referred to in the title, thankfully, have nothing to do with the actual hosting companies I'm dealing with, and are instead due to a foolish mistake on my part: when I switched my domain to WebFaction, I forgot that I had custom MX records enabling the use of my hosted google apps for my domain. As a result, as the new DNS information started propagating, people stopped being able to send me email. Fortunatly, it was an easy fix to just change the MX records with WebFaction, and I don't think I missed any important emails, but if anyone out there got a bounceback when sending me an email, that's why.

Currently I've got two projects hosted on WebFaction servers. So far, I really like them. As managed hosts go, they're probably the best I've worked with, and they certainly make life very easy when building Django powered sites.

Today I got an email from one of the clients whose project is hosted on WebFaction saying that their site is down. So I checked it out, and while I was able to access it, it was extremely slow, to the point where a less forgiving browser/LAN setup might cause it to time out. So I fired off a support ticket to WebFaction, and within a couple minutes, not only was the site back up to speed, but I was provided with a very good explanation for why my server was having problems.

Apparently there was an explosion at one of WebFaction's data centers this weekend. It took out power to the data center, but fortunately no one was hurt and none of the servers were damaged. Obviously, there have been some interruptions in service for the servers in that data center (which includes both of my WebFaction projects), but they've already gotten a significant number of the servers back online (though only one of mine).

Amazingly, this is actually the second time I've had a server taken out by an explosion at a data center. The first time was with a hosted Microsoft Exchange server with a hosting company in London.

It really sucks having sites down, especially critical ones (fortunately only one of the projects I have hosted with them is critical, and it's the one that's back up already), but as reasons for downtime go, you have to admit that an explosion is a pretty good one.

As you may recall, a while ago I got myself an account at MediaTemple with the idea that I'd move all my websites over to there. I had previously been using Dreamhost, but wanted something a little more high quality so that I could reasonably offer hosting services to some of my clients. MediaTemple seemed like a good way to go, and for the most part their service has been great.

Unfortunately, I have run into a few problems. Most importantly, despite spending a fairly significant number of man-hours working on it, I've been unable to get Django running on my (dv) server. Yes, they have a (beta) program that makes it easy to run Django on a (gs) account, but for my needs a (gs) simply won't do and I really don't want to have multiple accounts with them. The end result of this is that several of my web pages are still running on Dreamhost because they require Django (this blog actually is as well even though it's currently a WordPress blog, because I want to switch to something Django-based and it seems like an unreasonable hassle to migrate my WordPress blog to a new server only to then have to migrate it again to new software, especially as I'm currently holding onto my Dreamhost account for my Django-based pages anyway).

The issue is now coming to a bit of a breaking point. Why? Because I'm currently working on a pretty large Django-based website that will be going live in the next month or two. For the purposes of development, it's being hosted on WebFaction, which has been an amazing host. They make it incredibly simple to host a Django site, to the point that basically zero setup is required. But as we get closer to the point of going live, I've been considering what the hosting needs of the site will be going further, and how to best serve them.

The site is a redevelopment of an existing site, so we can get a pretty good idea of what the traffic numbers are going to look like. This will let us extrapolate the RAM and bandwidth requirements pretty well too. The issue, is that once the Django version of the site goes live, we're going to start to expand it. Thanks to the capabilities of Django, it's being developed with the potential for massive growth in mind. Specifically, it's using Django's Sites framework to allow for expansion to several sites. Currently there are only two, but the Django version will go live with 4 or 5, and there's the potential to expand far beyond that.

This means that we're going to require a pretty large number of (software) servers. There's the MySQL server running the back-end, an HTTP server for static content, and then an HTTP server for each site, and they're all going to be using up resources to different extents. Trying to find the best hosting solution for this sort of setup has led me to a couple of options:

1. Stay with WebFaction. A number of people have said they believe that WebFaction's shared hosting plans should be able to accomodate this. WebFaction provides a very good combination of ease of use and low-level access, their prices are good, and the way they have their hosting set up, it's extremely simple to add another Django install complete with its own Apache/mod_python instance. They also offer dedicated server, but I think there are probably better routes to go than with WebFaction's dedicated servers.
2. Switch to Slicehost. I've only just learned about Slicehost, but so far they look like a pretty sweet deal. For a very reasonable price you get a virtualized server running a Linux distro of your choice (you can choose from 8 right now) run on Xen. They claim not to oversell their servers, so you're guaranteed to actually get the full capacity that you pay for (unlike with budget hosts such as Dreamhost). And since you're getting your own virtualized host you have full root access. They basically have nothing preinstalled, so you can easily set it up in whatever configuration you want without having to deal with the vagaries of the anointed hosting package (Plesk on MediaTemple, I'm looking at you). I really like like look of them, and they've been getting good reviews. They're currently listed as the number 2 hosting company on Djangofriendly, behind only WebFaction. With my background in IT and Linux administration, the fact that I'd have to manage everything myself isn't enough to scare me away either. The fact that they let you choose your Linux distro really appeals to me as well, as I'm by far more familiar and comfortable with Gentoo than any other Linux distro. I only wish they offered FreeBSD slices, but the only reason they don't is technical, and once that issue is resolved it sounds like they plan on it. In a lot of ways, they're basically a very affordable colocation provider. The biggest issue, it sounds like, is that apparently communication between different slices counts against your bandwidth allotment (for both slices, presumably). This means that as the site I'm working on grows, if it spreads out to multiple slices (which it undoubtedly would and which I'd want it to do since that will give the added reliability of spreading across multiple physical machines) we'll basically be billed for database access from the sites that aren't on the slice with the database server.
3. Colocation. Colocation is basically the 800 lbs. gorilla in the room. It costs a lot more, but you get what you pay for. With colocation we'd have all the advantages of Slicehost (minus the low price, of course) plus the ability to expand more or less arbitrarily. We could have as many physical machines as we wanted running as much or as little of the site as we wanted. Provided we're willing to pay, of course. On top of that all the server management would again fall to me, but this time without some of the nice shortcuts that Slicehost offers. Essentially, colocation is alway the fall-back option. But hopefully one of the other two hosts can offer us a solution that's a little more balanced: we get less control, but more simplicity and ease of use for a greatly reduced price.

At the moment, I'm leaning towards sticking with WebFaction for now. I'm already very impressed with what they offer, and from the sound of things, they'll continue to be a more than adequate host as we expand. But I'm also definitely looking for input. If anyone has any suggestions or recommendations I'd love to hear them. In particular, any first-hand experience with hosting large Django sites with any of these solutions are most welcome.