As you may recall, a while ago I got myself an account at MediaTemple with the idea that I'd move all my websites over to there. I had previously been using Dreamhost, but wanted something a little more high quality so that I could reasonably offer hosting services to some of my clients. MediaTemple seemed like a good way to go, and for the most part their service has been great.

Unfortunately, I have run into a few problems. Most importantly, despite spending a fairly significant number of man-hours working on it, I've been unable to get Django running on my (dv) server. Yes, they have a (beta) program that makes it easy to run Django on a (gs) account, but for my needs a (gs) simply won't do and I really don't want to have multiple accounts with them. The end result of this is that several of my web pages are still running on Dreamhost because they require Django (this blog actually is as well even though it's currently a WordPress blog, because I want to switch to something Django-based and it seems like an unreasonable hassle to migrate my WordPress blog to a new server only to then have to migrate it again to new software, especially as I'm currently holding onto my Dreamhost account for my Django-based pages anyway).

The issue is now coming to a bit of a breaking point. Why? Because I'm currently working on a pretty large Django-based website that will be going live in the next month or two. For the purposes of development, it's being hosted on WebFaction, which has been an amazing host. They make it incredibly simple to host a Django site, to the point that basically zero setup is required. But as we get closer to the point of going live, I've been considering what the hosting needs of the site will be going further, and how to best serve them.

The site is a redevelopment of an existing site, so we can get a pretty good idea of what the traffic numbers are going to look like. This will let us extrapolate the RAM and bandwidth requirements pretty well too. The issue, is that once the Django version of the site goes live, we're going to start to expand it. Thanks to the capabilities of Django, it's being developed with the potential for massive growth in mind. Specifically, it's using Django's Sites framework to allow for expansion to several sites. Currently there are only two, but the Django version will go live with 4 or 5, and there's the potential to expand far beyond that.

This means that we're going to require a pretty large number of (software) servers. There's the MySQL server running the back-end, an HTTP server for static content, and then an HTTP server for each site, and they're all going to be using up resources to different extents. Trying to find the best hosting solution for this sort of setup has led me to a couple of options:

1. Stay with WebFaction. A number of people have said they believe that WebFaction's shared hosting plans should be able to accomodate this. WebFaction provides a very good combination of ease of use and low-level access, their prices are good, and the way they have their hosting set up, it's extremely simple to add another Django install complete with its own Apache/mod_python instance. They also offer dedicated server, but I think there are probably better routes to go than with WebFaction's dedicated servers.
2. Switch to Slicehost. I've only just learned about Slicehost, but so far they look like a pretty sweet deal. For a very reasonable price you get a virtualized server running a Linux distro of your choice (you can choose from 8 right now) run on Xen. They claim not to oversell their servers, so you're guaranteed to actually get the full capacity that you pay for (unlike with budget hosts such as Dreamhost). And since you're getting your own virtualized host you have full root access. They basically have nothing preinstalled, so you can easily set it up in whatever configuration you want without having to deal with the vagaries of the anointed hosting package (Plesk on MediaTemple, I'm looking at you). I really like like look of them, and they've been getting good reviews. They're currently listed as the number 2 hosting company on Djangofriendly, behind only WebFaction. With my background in IT and Linux administration, the fact that I'd have to manage everything myself isn't enough to scare me away either. The fact that they let you choose your Linux distro really appeals to me as well, as I'm by far more familiar and comfortable with Gentoo than any other Linux distro. I only wish they offered FreeBSD slices, but the only reason they don't is technical, and once that issue is resolved it sounds like they plan on it. In a lot of ways, they're basically a very affordable colocation provider. The biggest issue, it sounds like, is that apparently communication between different slices counts against your bandwidth allotment (for both slices, presumably). This means that as the site I'm working on grows, if it spreads out to multiple slices (which it undoubtedly would and which I'd want it to do since that will give the added reliability of spreading across multiple physical machines) we'll basically be billed for database access from the sites that aren't on the slice with the database server.
3. Colocation. Colocation is basically the 800 lbs. gorilla in the room. It costs a lot more, but you get what you pay for. With colocation we'd have all the advantages of Slicehost (minus the low price, of course) plus the ability to expand more or less arbitrarily. We could have as many physical machines as we wanted running as much or as little of the site as we wanted. Provided we're willing to pay, of course. On top of that all the server management would again fall to me, but this time without some of the nice shortcuts that Slicehost offers. Essentially, colocation is alway the fall-back option. But hopefully one of the other two hosts can offer us a solution that's a little more balanced: we get less control, but more simplicity and ease of use for a greatly reduced price.

At the moment, I'm leaning towards sticking with WebFaction for now. I'm already very impressed with what they offer, and from the sound of things, they'll continue to be a more than adequate host as we expand. But I'm also definitely looking for input. If anyone has any suggestions or recommendations I'd love to hear them. In particular, any first-hand experience with hosting large Django sites with any of these solutions are most welcome.

So, a little while ago I got a MediaTemple (dv) server. In general, they seem to be pretty good. The server's been fast and reliable so far, I get a good amount of storage and bandwidth for the price, and I have root access so theoretically I can do pretty much whatever I want. I have run into a bit of a problem however.

For the past several days I've been banging my head against installing Django on my MediaTemple server. In theory this should be pretty easy. The server comes with Apache and mod_python installed, so all that I should need to do is check out the lated django trunk from svn, make a few symlinks, install MySQLdb, change some Apache configurations to tell it to use Django for the appropriate URLs, and go. In practice, it's not working quite so well.

The first few steps were simplicity itself. It wasn't long before I had a successful Django install and was able to 'python manage.py syncdb' to have my projects database tables created. Changing the Apache settings was a little more difficult because you can't actually modify the httpd.conf as Plesk will overwrite it. Of course, there's no documentation telling you this or telling you the appropriate solution (despite, I might add, the insistence of the MediaTemple KnowledgeBase that instructions for installing Django on a (dv) server exist somewhere). A little help from Django's Google Groups community helped me out with that, though, letting me know that I instead need to create a /var/www/vhosts/MYDOMAIN.COM/conf/vhosts.conf file and put the settings in there. So I did that, and it even seems to work, but when I try to actually go to the page I get this:

Mod_python error: "PythonHandler django.core.handlers.modpython"


Traceback (most recent call last):

  File "/usr/lib/python2.3/site-packages/mod_python/apache.py", line 299, in HandlerDispatch
    result = object(req)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/modpython.py", line 188, in handler
    return ModPythonHandler()(req)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/modpython.py", line 161, in __call__
    response = self.get_response(request)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/base.py", line 64, in get_response
    response = middleware_method(request)

  File "/usr/lib/python2.3/site-packages/django/contrib/sessions/middleware.py", line 15, in process_request
    request.session = engine.SessionStore(session_key)

AttributeError: 'module' object has no attribute 'SessionStore'

Apparently, for some reason, there's some problem with Django's sessions middleware, although no one else seems to have discovered this problem. If I remove sessions (and therefore the admin app) from my project the page will actually load, but it then fails to actually be able to get anything from the database so gives me errors on any page that requires database calls (most of them). Thus far neither the Google Groups nor the expert aid of Jesse Legg have been able to help me make any progress in solving this issue.

There are people out there running Django on MediaTemple (dv) servers, so I know this is possible, and I'm confident that I'll get it eventually. But in the mean time it's incredibly frustrating and is putting serious delays in the process of transitioning my sites over to MediaTemple (not to mention in the development of some other Django projects I'm working on right now). You can be sure that when I finally get Django working on my (dv) server that I'll be writing a detailed account of exactly how it was done. There really needs to be some documentation on this out there.