Yes, I realize that this is now my third post not related to DVCSes since I said my next post would be about DVCSes. So sue me.

I recently encountered an interesting requirement for the Django admin: we wanted people to normally only see (and be able to edit) objects that either they created themselves, or that the creator had assigned them to. Fortunately this is insanely easy in Django 1.1, all you have to do is override the queryset() method of the appropriate ModelAdmin like so:

This snippet very easily allows you to apply essentially any filter you want to the QuerySet that gets passed on to the change_list and allows you to provide an exception for super users (always a good thing!). That part was really easy and something I've done before (even in 1.0.x where the functionality is there, just not exposed). Where it got trickier was where the client also wanted the functionality to be able to view all the objects regardless of who created them, but still only have editing capabilities for their own objects (and only see the information available on the change_list for others).

This part was much harder, but fortunately also made very possible by the updates to ModelAdmin in the 1.1.x branch. The first thing I wanted to do was just provide a new URL and view integrated seamlessly into the admin. Again this was very simple with the new admin, and required only overriding the get_urls() method on the ModelAdmin:

Getting that all_objects view to return something essentially identical to the normal change_list, but with a completely different filter on the queryset, and some different display options was the real problem I had to address. Wrapping my head around the problem took some time, but fortunately even this was pretty simple once I really started to get into the flexibility of the ModelAdmin class.

Broken down to it's most basic level, what I wanted to do was return the change_list view from a Model Admin. This in itself is very simple to do, and requires very little code:

Once I figure that out it was pretty obvious that what I needed to do was subclass my ModelAdmin and just re-overide the relevant functions. Turns out this is really easy to do, and gives you a whole lot of flexibility. So I created a special AllItemsAdmin subclass of ThisAdmin, and overrode that queryset() method to return all the objects. I then had to figure out a way to get it to only display but not link to edit pages for objects that the current user doesn't own. Since I needed them to be in the queryset, this was a little trickier.

Anyone who's been working with the Django admin should know about the list_display option on the ModelAdmin. What you might not know (I didn't until recently) is that no only can the list_display list contain field names and properties from the model (Actually, if you didn't know that you can provide callable properties from your model, you should check that out. It lets you do some very useful things.), you can also provide callables from within your model admin. So you could set your list_display to something like ['title', 'my_function'] where my_function is a method on your model admin with a definition along the lines of my_function(self, object) that can perform operations on the Django object for that row and return whatever value you want. The normal options for model properties (such as allow_tags and short_description) work here as well. So what I was able to do with this was create a custom column for my change_list that looked at the specific object, checked the ownership properties as above, and then returned either just the name of the object, or the name of the object as a link to the regular edit page for that object. By setting list_display_links to (None,) I was able to prevent any of the fields from automatically be turned into links. Of course doing this required that method to have access to the request which it usually wouldn't, but since I was already working with a hacked up subclass of my ModelAdmin I was able to just override the __init__() to take the request object and pass that in when I instantiated it. What I ended up with was this really awesome view (if I do say so myself):

As you can see, this is also using another new feature in the 1.1.x admin: reversing of admin URLs. After this all I needed was a few simple changes to the change_list.html template for this model let me add a 'View all' link to go to this new view, and then the 'all' context variable being passed in as extra context simply tells it to provide a link back to the standard view otherwise.

The result of all this was a seamless integration of my custom 'view, but don't edit all' view into the Django admin.

I just happened to check my Google analytics today and saw that my hits had dropped essentially to zero since October 8th. Turns out that my wsgi file was a little screwed up after my initial attempts to migrate my blog to django-mingus. I have, obviously, fixed it, though I still want to move the blog to mingus and will hopefully get the chance to actually do that soon. Until then, however, at least my blog is up!

I just saw this very interesting (for a programmer) blog entry on Storing IP addresses as integers. As the article says, anyone who wants to store IP addresses in a database is generally going to do as a string. However if you're really concerned about memory efficiency, you might want to find a lighter data type to use. Since an IP address is really just a collection of integers it would make sense to store it as an integer, however doing so without losing important information (specifically, the placements of the dots) becomes an issue.

The blog post in question introduces the the pack() and unpack() functions, which 'allow you to create and extract data into and out of binary-packed strings' and provides the Ruby code necessary to encode an IP address as a simple integer and decode it back to a dotted quad. I thought this was pretty cool, so I decided to write the equivalent Python code. This is what I ended up with:

It seems to work as advertised, though the packed string the encode() function returns is different from what Ruby gives you, so they won't interoperate (it would probably only require changing the pack() and unpack() formats to do so, but I didn't feel like experimenting to figure out which ones).

Of course, being a Django guy, the immediate application of these functions that I see is creating a new IPAddressField that stores the address as an integer instead of a string transparently.

Discovery Creative has a lot of web sites and apps out there. And quite a few of them need to send email in one form or another. Previously that has meant that every single one of those sites/apps needed to implement it's own mechanism for sending email. This is obviously a bit of a pain, not to mention a potential security risk, and a blatant violation of the DRY principle. So I was tasked with building a better mousetrap, as it were.

One of the cool new things I had heard about at DjangoCon 2009 (which I really should have written about...) was Piston, a framework built in Django for building RESTful APIs by the bitbucket guys. I'd never actually built any sort of API before, but it seems to be the thing to do, so I decided to take django-piston for a whirl and how it works. As it turns out, it works extremely well. So well, in fact, that when coupled with some of the fun tools that Django provides (such as ModelForms) you can easily build a RESTful API in no time at all.

Thanks to django-piston I was able to create a simple API that will allow us, moving forward, to use a single, centralized email solution for all our web apps. In fact anything that can send an HTTP POST request (including curl, which is what I've been using for testing) can send email using the API I created so long as it can also handle HTTP authentication (which django-piston easily handles against django.contrib.auth). I'm hoping, after a little more work and refinement, to open-source our API so that other can benefit from it (and, hopefully, contribute back to it!), but for now you'll have to make do with a sample project that I threw together for the most recent django-district meeting this past Thursday: Django-Piston Presentation. It's a bitbucket repository that includes all the code for a simple RESTful API that allows you to create, fetch, and delete objects from a simple Django object. This particular project was designed to be extremely flexible, and all one needs to do is add or remove fields from the model (or point it at a different model) to adapt it to just about anything. Hopefully it will serve as a pretty good instructional example to anyone who wants to create their own API with django-piston. The project also contains my notes on how to build and test it in an Emacs org-mode file.

Next up: Git, Mercurial, and moving on from Subversion.

Just a quick note: this blog now has Disqus comments. Turns out it's insanely simple to implement, thanks to Arthur Koziel's awesome django-disqus app. Took me no more than five minutes during my lunch break to get everything to it's current state.

One of the reasons that I haven't been blogging lately is that I have a lot of planned changes for the blog. Most importantly, I've been planning on moving it to my Cloud Server on The Rackspace Cloud (née Mosso). I've finally taken care of that, and this post is the first on the new server. If you can see it, that means the DNS changes have propagated.

Now that I've done this I have a number of other changes planned. I'm going to completely overhaul the backend to the blog. In the course of the migration I upgrade from Django 1.0.2 to Django 1.1. Nothing else has changed yet, but I intend to also revamp the templates, add some fancy new feature to better integrate with the rest of my online life, and start using django-mingus. One of the more urgent changes I want to make is to switch to Disqus powered comments. Previously I was using django.contrib.comments along with Akismet for spam filtering. Akismet really just hasn't worked at all for me for the past few months, which is why comments are current disabled. Fortunately, Disqus should cut down on the spam by requiring registration without imparting too large an onus on the readers by using a number of common authentication backends like OpenID, Facebook Connect, and Twitter. I'm hoping it will make for a good compromise.

The most important change, of course, is that I intend to actually start writing again. I've got a lot of ideas that I've wanted to write about and simply haven't because either I didn't have the time, I wanted to wait until this migration, or I was just too lazy. Hopefully, that will be changing now!

I will get around to actually writing something soon, I swear! But for now you'll all just have to make do with another announcement of a project going live. This time it's actually up on Discovery Channel site. It's a Flash and Django based app promoting one of Discovery's new shows The Colony, about survival in a post-apocalyptic LA. I've actually been looking forward to this show, can't wait to watch it!

I'm actually a bit late on this one, but I've been so busy with my next Discovery Creative project that I just haven't had time for much else. Anyway, from June 15 through 22 AFI, in partnership with the Discovery Channel will be putting on their Silverdocs Documentary Film Festival. Check out the website for more details on the festival and the films they'll be showing, not to mention an example of how awesome my (and the rest of Discovery Creative, I guess) work is.

I can't believe I haven't written about this yet, but BostonChefs.com's the unofficial guide to Boston's Restaurant Week for Winter 2009 is live. I've been working on this particular site since 2007 when I first help make some PHP-based improvements to it. Last summer it became my first professional Django-based project when I completely redeveloped it in Django. This year I've made some further improvements to the Django codebase, including a completely re-implemented and much improved Google Maps mashup that lets you view the restaurants geographically.

Previously this had been using some old code that I had inherited from the original site. It was pretty good code, but more modern tools allowed me to vastly simplify it. Specifically, instead of always loading every restaurant and populating the map with markers, then simply zooming in on a specific neighborhood or restaurant, it now only loads the restaurants appropriate for the current target. If you select a specific neighborhood, it only loads the restaurants for that neighborhood, and puts markers on the map for them. Because of that I was able to use Google's API to automatically set the bounds and zoom level of the map to best show those restaurants. This removes the need to manually pick a set of coordinates and zoom level for each neighborhood (having, then, to fine tune it for each neighborhood individually) and instead takes care of all that automatically. The result is a map that's simpler to manage, presents the relevant information more intelligently, and is much quicker to load. All in all, I think it's a pretty big improvement.

I just finished up with some relatively simple, but still fun modifications to the Django admin site for one of the projects I'm working on. For this project I needed to create a many-to-many relationship between two models with ordering information associated with it. This is fairly easy to accomplish by creating a many-to-many with an intermediary table. But providing a convenient mechanism for managing that information is a little trickier. By default you'll just end up with a text entry box in which to manually type the order for that item. This gets pretty old pretty fast, so finding a better method is important. In the past I've used jQuery to add drag and drop re-ordering to inline edited models, but this time I needed to do a little more as well. Specifically I wanted to make it easier to add more inline item. Ordinarily you just set an arbitrary number of empty boxes to have displayed (by default 3) and if you want more you have to fill those boxes, then hit 'Save and continue editing' to get three more. This is a pretty crappy way to do it (but the only way without introducing unwanted dependencies).

Some googling revealed that Arne Brodowski had done pretty much exactly what I wanted to so, so I worked up some modified version of his scripts. The first step was setting it up to hide entries that were marked for deletion. Arne provided a prototype script to do this, but I made a few modifications that clean it up a bit and made it actually work (at least with jQuery 1.3.1). What I ended up with was this somewhat more elegant looking script:

jQuery(function($) {
	$("div.inline-related input:checkbox[id$=DELETE]").change(function() {
		if ($(this).attr('checked')) {
		    $(this).parents('div.inline-related').children('fieldset.module').addClass('collapsed');
		} else {
		    $(this).parents('div.inline-related').children('fieldset.module').removeClass('collapsed');
		}
	    });
    });

The changes I made are mainly in the selector for grabbing the checkboxes ($("div.inline-related input:checkbox[id$=DELETE]"), and in the method for checking whether or not the box is actually checked ($(this).attr('checked')). With those changes it works exactly as advertised, which is a pretty handy bit of functionality.

The next step was slightly more complex. To be able to dynamically add more relationships without having to 'Save and continue editing', I made basically the same template changes as Arne did, and fortunately this time didn't need to make many changes to the script. I basically just removed the 'return false' from the end and wound up with this:

function increment_form_ids(el, to, name) {
    var from = to-1  ;  
    $(':input', $(el)).each(function(i,e){
	    var old_name = $(e).attr('name');
	    var old_id = $(e).attr('id');
	    $(e).attr('name', old_name.replace(from, to));
	    $(e).attr('id', old_id.replace(from, to));
	    $(e).val('');
	});
}


function add_inline_form(name) {
    var first = $('#id_'+name+'-0-id').parents('.inline-related');
    var last = $(first).parent().children('.last-related');
    var copy = $(last).clone(true);
    var count = $(first).parent().children('.inline-related').length;
    $(last).removeClass('last-related');
    $(last).after(copy);
    $('input#id_'+name+'-TOTAL_FORMS').val(count+1);
    increment_form_ids($(first).parents('.inline-group').children('.last-related'), count, name);
    $(first).parents('.inline-group').children('.last-related').find('input[id$=order]').val(0);
    $('div.inline-group').find('div.inline-related').each(function(i) {
	    $(this).find('input[id$=order]').val(i+1);
	});
}

You'll probably notice that I did actually make one other change. After I had gotten all this working, I realized there was one problem: If when starting from scratch you just happened to enter your choices in the order you wanted it wouldn't actually save that ordering information. By default (with my model definitions anyway) everything was assigned an order of 0 until you actually dragged things around to reorder them. This might be ok for some applications, but just won't work for this particular project. So I added those three lines of code to the function for adding new entries and also made sure that it's also done when you first load the page. This way you can be sure that everything will always be numbered appropriately and you won't end up with any unwanted 0s in your ordering information.

I recently went live with another Django project. The Small Design Firm website uses Django and Prototype frameworks to showcase the company's previous projects and news articles. This was a pretty straightforward project and didn't really involve any real hacking, but was none the less a good reminder of how pleasant it is to work in Django and how quickly one can develop a website with it.

This post is actually a few weeks overdue, but we were away on vacation for a couple weeks, and I'm still in the process of catching up from everything. Anyway, the new BostonChefs.com is live! There haven't been any major changes since the public beta I announced a few months ago, but there have been a few minor tweaks and even a couple new features added. This is my first major Django project to go live, and while there's still some work to be done it's nice to see the product of so much work finally going out the door.

But there's no rest for the weary, and I've got a fairly large number of more projects in the pipeline including at least three more Django projects of varying sizes, and one in CodeIgniter. The next few months, if not the whole of 2009, promise to be quite busy!

As you probably know, I've been working on a Django-based re-build of BostonChefs.com (the new version of which is actually live now, but due to DNS propagation issues isn't yet available to 100% of people which is why I haven't yet written a post about it). Among other things, BostonChefs.com provides information on some of the fantastic restaurants in the Boston area. One piece of information it provides is the hours of operation of those restaurants. In order to store this information I created a model called HoursOfOperation. It looks like this:

class HoursOfOperation(models.Model):
    DAY_CHOICES = (
        ('0', 'Sun'),
        ('1', 'Mon'),
        ('2', 'Tue'),
        ('3', 'Wed'),
        ('4', 'Thur'),
        ('5', 'Fri'),
        ('6', 'Sat'),
        )
    restaurant = models.ForeignKey("Restaurant")
    meal_period = models.ForeignKey("MealPeriod")
    day = models.CharField(max_length=3, choices=DAY_CHOICES)
    open_time = models.TimeField(default=datetime.datetime.now)
    close_time = models.TimeField(default=datetime.datetime.now)


    def _get_hours(self):
        return "%s - %s" % (self.open_time.strftime('%I:%M%p'), self.close_time.strftime('%I:%M %p'))
    hours = property(_get_hours)

As you can see, each 'hour' is related to a restaurant and a meal period, which allows us to display the information in a manner similar to that you might find on a store's front sign. For example, if you go to the Grill 23 & Bar page (my personal favorite restaurant in Boston, although Craigie on Main is a decent challenger), you'll see something like this:

DINNER
* Sun: 5:30 p.m.-10 p.m.
* Mon-Thur: 5:30 p.m.-10:30 p.m.
* Fri: 5:30 p.m.-11 p.m.
* Sat: 5 p.m.-11 p.m.

Building a list like that out of the above model proved slightly more difficult that I might have hoped. It required quite a lot of template logic, including writing a custom filter. The block of template code necessary to generate that list looks like this:

<div class="hours">
    {% regroup restaurant.hoursofoperation_set.all by meal_period as periods %}
    {% for period in periods %}
    <div class="hoursMealPeriod">{{ period.grouper }}</div>
    {% regroup period.list by hours as hour_list %}
    <ul>
        {% for hour in hour_list %}
        <li>{{ hour.list|collapsedays }}</li>
        {% endfor %}
    </ul>
    {% endfor %}
</div>

As you can see, somewhat complex. Those nested {% regroup %}s can be nasty to wrap your head around, if nothing else. But basically it's taking the set of HoursOfOperation objects related to the restaurant, grouping them by meal period, then taking the subset of those objects for each meal period, and grouping those by the hours of the day they represent. So what you're then left with is a list of all the different time periods (still represented as HoursOfOperation objects) that the restaurant is open for a given meal period, and the days on which it is open during those hours. As you can see above, the days are represented by number of the day of the week (0 for Sunday through 6 for Saturday).

Converting that list integers into something like 'Mon, Wed-Fri' was not very easy, and certainly not something I wanted to try to tackle using Django's template tags. I ended up drawing heavily on my hazy memories of CS 127 (many thanks to Dave who taught me all about recursion way back then) and creating a filter that considers the list of HoursOfOperation objects as a list of those integers, then recursively converts it into a list of lists representing the subsets of contiguous days in the list. So if you start out with [1, 3, 4, 5] you end up with [[1, 1], [3, 5]] which is then converted into 'Mon, Wed-Fri'. After several false starts I ended up with this beauty of a Django template filter:

from django.template import Library
from django.template.defaultfilters import time


from types import ListType

register = Library()

def simplify(index, found, days):
    high = index+1
    mid = index
    low = index-1
    if not found:
        days[low] = [days[low], days[low]]
    if high >= len(days):
        if not isinstance(days[-1], ListType):
            if days[-1] == days[-2][1]:
                days.pop(-1)
            else:
                days[-1] = [days[-1], days[-1]]
        return days
    if int(days[high].day) - int(days[mid].day) == 1 and (found or int(days[mid].day) - int(days[low][0].day) == 1):
        days[low][1] = days[high]
        days.pop(mid)
        high = high-1
        found = True
    else:
        if found:
            days.pop(mid)
        found = False
    return simplify(high, found, days)

@register.filter
def collapsedays(value):
    hours = "%s-%s" % (time(value[0].open_time), time(value[0].close_time))
    days = simplify(1, False, value)
    for i in range(len(days)):
        if days[i][0] == days[i][1]:
            days[i] = days[i][0].get_day_display()
        else:
            days[i] = "%s-%s" % (days[i][0].get_day_display(),
                                 days[i][1].get_day_display())
    return "%s: %s" % (', '.join(days), hours)

Really, this was a pretty major oversight on my part, but I just now finished added a 'status' field to each entry in my blog app. It's a pretty simple thing in and of itself, I just added a tiny bit of code to my Entry model:

STATUS_CHOICES = (
        ('dr', 'draft'),
        ('ac', 'active'),
        ('ar', 'archive'),
    )


status = models.CharField(choices=STATUS_CHOICES, max_length=2, default='dr')

And then a custom manager as well:

class EntryManager(models.Manager):
    def active(self):
        return self.get_query_set().filter(status='ac')

Remembering, of course, to make sure that manager was actually accessible from the model (in this case by adding 'objects = EntryManager()' to my model class).

Once I did that I was able to change the Entry.objects.all() in my view(s) to Entry.objects.active() and now I'm able to write a post and save it without actually publishing! As I said, something of a bonehead move that I didn't do this in the first place.

Since I was messing around in my code anyway I decided to clean up the admin for my Entry model as well. Since I'm the only one who ever uses the admin for my blog, I hadn't bothered to before, but now I've got a little more info available to me when looking at my entries:

class EntryAdmin(admin.ModelAdmin):
    prepopulated_fields = {'slug': ('title',)}
    save_on_top = True
    list_display = ['title', 'post_date', 'site_list', 'status',]
    list_filter = ('sites', 'status',)

So my admin now looks something like this:

A cool thing there is the 'site_list' column. By default, you can't use a ManyToManyField in the list_display for your model. But I've dealt with this before, and it's only 5 lines of code to make this work:

def site_list(self):
        if self.sites:
            sites = [site.name for site in self.sites.all()]
            string = ", ".join(sites)
            return string

Good stuff!

Every so often you run into a situation where Django's built-in form fields and widgets just don't meet your needs. I ran into this the other day when creating a credit card processing form. I wanted an easy way for the user to enter the expiration month and date of their card, but the tools provided by django only gave me a single text field and the option to use a javascript date picker. Neither of those was quite what I wanted, I just wanted two selects that would allow the user to pick the month and the year, as on pretty much every credit card form you've ever filled out online.

The obvious option would be to make two separate fields on your model, one for month, and one for year. But I don't really like that option. I would much rather have both selects show up on the same line, which is not the behavior you'd get with two separate fields. So I decided to write a custom widget and field to accomplish this. It was actually surprisingly easy to do. Probably the most difficult part was coming up with a decent way to populate the selects with worthwhile options. I'm not entirely pleased with the route I took there, but it's easy enough to change later. Here's my code:

from django import forms


import datetime

class MonthYearWidget(forms.MultiWidget):
    """
    A widget that splits a date into Month/Year with selects.
    """
    def __init__(self, attrs=None):
        months = (
            ('01', 'Jan (01)'),
            ('02', 'Feb (02)'),
            ('03', 'Mar (03)'),
            ('04', 'Apr (04)'),
            ('05', 'May (05)'),
            ('06', 'Jun (06)'),
            ('07', 'Jul (07)'),
            ('08', 'Aug (08)'),
            ('09', 'Sep (09)'),
            ('10', 'Oct (10)'),
            ('11', 'Nov (11)'),
            ('12', 'Dec (12)'),
            )

        year = int(datetime.date.today().year)
        year_digits = range(year, year+10)
        years = [(year, year) for year in year_digits]

        widgets = (forms.Select(attrs=attrs, choices=months), forms.Select(attrs=attrs, choices=years))
        super(MonthYearWidget, self).__init__(widgets, attrs)
    
    def decompress(self, value):
        if value:
            return [value.month, value.year]
        return [None, None]

    def render(self, name, value, attrs=None):
        try:
            value = datetime.date(month=int(value[0]), year=int(value[1]), day=1)
        except:
            value = ''
            
        return super(MonthYearWidget, self).render(name, value, attrs)

class MonthYearField(forms.MultiValueField):
    def compress(self, data_list):
        if data_list:
            return datetime.date(year=int(data_list[1]), month=int(data_list[0]), day=1)
        return datetime.date.today()

Simple, no? The results end up looking like so:

Django provides an extraordinarily easy way of integrating file uploads into your site in the FileField (also the ImageField which provides some nifty image-specific functionality). However as soon as you start dealing with file uploads you have to worry about where those files are going to be stored. To take care of this, the 'upload_to' argument is set when declaring your FileField which is a local filesystem path that will be appended to your MEDIA_ROOT to the upload location (you could also use an absolute path). By default this is just a static string, though it does handle strftime arguments to allow some flexibility and will look something like this:

def model(models.Model):
    file = models.FileField(upload_to='my/file/uploads/')
    ...

However that's not always going to cut it. Sometimes you're going to want more flexibility that even just separating the files out by date information. In that case you can take advantage of the fact that a callable can be passed as an argument. In one of my projects I've defined a function that returns the appropriate path depending on a couple of different variables. In this particular case, I know that all uploads for a certain model will be PDFs, while all uploads for certain other models will be images, so I'm able to pretty easily generate appropriate paths for those uploads. We also wanted to obfuscate the filenames so that someone couldn't easily guess the path for other files, and didn't really like the way that Django simply keeps appending '_'s to filenames in the event of a filename collision until it ends up with a unique name. To handle that I simply generate a random string to use as the filename, which fits our needs quite nicely. In my specific case, the function lives in myproject.lib.files and looks like this:

import random, string
from django.contrib.contenttypes.models import ContentType


def get_path(instance, filename):
    ctype = ContentType.objects.get_for_model(instance)
    model = ctype.model
    app = ctype.app_label
    extension = filename.split('.')[-1]
    dir = "site"
    if model == "job":
        dir += "/pdf/job_attachment"
    else:
        dir += "/img/%s" % app
        if model == "image_type_1":
            dir += "/type1/%s" % instance.category
        elif model == "image_type_2":
            dir += "/type2"
        elif model == "restaurant":
            dir += "/logo"
        else:
            dir += "/%s" % model
    
    chars = string.letters + string.digits
    name = string.join(random.sample(chars, 8), '')
    
    return "%s/%s.%s" % (dir, name, extension)

As you can see, it's pretty simple to generate an upload path based on pretty much any characteristics you want. To then tell Django to use this function to get the upload path rather than a static string you modify your model definition like so:

from myproject.lib.files import get_path


def model(models.Model):
    file = models.FileField(upload_to=get_path)
    ...

And that's all it takes. Now all my uploads are automatically renamed and uploaded into the particular directory structure that we want. And since all the logic involved is in a single function it's extremely easy to change it at any point if we decide to alter our directory structure.

One of the Django projects that I've been working on for about a year and and will (fingers crossed) be going live in the very very very near future has involved a lot of modification to Django's admin interface. I plan on writing more about the many, many specific changes that I've made to the interface (without modifying the actual Django codebase, so that the changes can be easily applied by anyone without breaking updates), but to talk about them all at once would make far too long of a post. So I'll be taking them on one at a time.

The change I want to talk about right now involves redirecting the user to the page I want them to be at after they've finished adding or editing an object rather than to that object's model's change_list. Normally, if you're editing an Entry object in the Blog app, when you hit save it will take you to /admin/blog/entry/, which is a list of all the Entry objects in the database. However there are some instances in which this isn't the behavior I want. Once such instance involves model inheritance. Say, for example, you have multiple types of Entries which you've accomodated through multi-table-inheritance. Because the different sub-classes are different models, they all have their own change_lists in the Django admin. But I want to be able to view, edit, and create Entries of all types from one page.

Fortunately, Django makes this fairly easy to accomplish. All that is necessary is to override the appropriate methods in the Entry ModelAdmin. That will end up looking something like this:

class EntryAdmin(admin.ModelAdmin):
    def change_view(self, request, object_id, extra_context=None):
        result = super(EntryAdminAdmin, self).change_view(request, object_id, extra_context)


        if not request.POST.has_key('_addanother') and not request.POST.has_key('_continue'):
            result['Location'] = iri_to_uri("/admin/blog/entry/")
        return result

The exact same modification should be made to add_view as well, and a nearly identical modification to delete_view though it doesn't need to deal with the _addanother and _continue cases. You can then use the EntryAdmin class for all of your varioud Entry sub-classes, or, if you need some other changes to the admin for different Entries sub-classes you can sub-class EntryAdmin for them. Now, whenever you hit the save button after editing any sort of Entry, it will always take you back to /admin/blog/entry/ rather than /admin/blog/linkentry/ or whatever your other subclasses are. If you want it to only take you back to /admin/blog/entry/ if you're coming from a particular page and otherwise take you to /admin/blog/linkentry/ all you need is to add a GET variable to your url (something like '/admin/blog/linkentry/add/?return_to_main=True') and then check for it in your modified change_view, add_view, and delete_view methods with a request.GET.get('return_to_main', False). I've even used this between objects of different model types to create a 'dashboard' page that allows you to view, and alter the relationships between an object of one type with objects of another type. All that's necessary in a case like that is to pass the id of the object in your GET variable and take that into account when creating your uri. An added benefit of that it makes it easy to auto-fill the ForeignKey field when creating a related object. In such a case you'll also need to keep that GET variable in the URLs down the line in order to maintain compatilibility with the 'Save and add another' and 'Save and continue editing' features. But that's still a simple modification:

class EntryAdmin(admin.ModelAdmin):
    def change_view(self, request, object_id, extra_context=None):
        result = super(EntryAdminAdmin, self).change_view(request, object_id, extra_context)


        other_id = request.GET.get("other_id", None)
        if not request.POST.has_key('_addanother') and not request.POST.has_key('_continue'):
            if other_id:
                result['Location'] = iri_to_uri("/admin/dashboard/%s/") % other_id
                return result
        elif request.POST.has_key('_continue'):
            if other_id:
                result['Location'] = iri_to_uri("?other_id=%s" % other_id)
                return result
        elif request.POST.has_key('_addanother'):
            if other_id:
                result['Location'] = iri_to_uri("%s?other_id=%s" % (result['Location'], other_id))
                return result
        return result

But more on that sort of thing in other posts.

I've been nothing but impressed with the service I've got from WebFaction and the reliability I've gotten from their servers. I even had a very high traffic site run on a WebFaction shared account without a hitch.

Today, however, we got a first hand look at the downside of a shared server. If you're a webfaction customer you should be (no really, you should) subscribed to their status blog It's a great way to be kept up to date on any issues that might affect your site. The most recent issue has to do with the MySQL server on web49: the server that we happen to be using to develop a very large project. As you can see from reading the entry, the problem appears to have been caused by a corrupted database table (not one of ours) which was causing some unreliability with the database server (our Django based site was intermittently unable to connect to the database and, when it could, intermittently unable to authenticate). Though they thought they had it fixed, the problem returned and while they're attempting to fix it for good they've rolled back the entire database server to a known good backup.

This is a good approach as it means that everyone should still have most of their data in the meantime. Unfortunately, we happen to be in the middle of populating the database with the data we need to go live in the near future. Rolling back the database even by a day means that we've lost a ton of work. We should get it all back once the problem is fixed, but of coruse that means that we have to put the work on hiatus until the problem is fixed to avoid versioning issues.

This right here is the perfect illustration of why a dedicated server is a good idea. Yes, a shared server might be able to support your site. But it also leaves you vulnerable to the actions of the people you're sharing the server with. If someone else does something stupid that corrupts a database table on a server that they share with you suddenly you stand to lose a lot of work. If someone has a poorly written app that somehow manages to crash the server or even just eat up all the RAM, your site goes down. With a shared server you simply don't have the security of knowing that your site is stable and secure even if you trust your hosting company and you trust your code.

That security is what you're paying for when you get a dedicated server over a shared one.

I've been using WebFaction for my hosting for a while now, and have been extremely pleased with them. In addition to the fantastic service I've received, I've been very impressed with the intelligent way they have their servers setup. They've clearly done a lot of work to make things as modular as possible which makes it insanely easy for me to run multiple sites with very different requirements seamlessly on the same server.

Basically what they've done is segment out all of your different websites into 'applications'. Each application in represented as a directory in your ~/webapps/ directory, and is essentially a self-contained environment with it's own apache instance, and, in the case of a Django app, it's own $PYTHONPATH. The end result is that even though all the websites are being stored and run from within my home directory, they're entirely modular, can have different, or different versions of the same, dependencies installed, and can be shut down and restarted independently of one another. On top of all this is a fantastically simple custom web-based control panel that I'm pretty sure is built with Django.

I've been so impressed with how well this setup works, that I've decided to duplicate it on my home server for development purposes. Currently I do pretty much all my development work on my Gentoo Linux powered ThinkPad. To that end I've installed Apache, MySQL, PostgreSQL, SQLite, Python, PHP, &c.; to allow me to mimic the live sites as closely as possible and to allow me to continue working when I don't have internet access (such as when I'm flying or visiting Jessi's family out beyond the reach of broadband). This works very well, but as I'm just using a basic Apache install, without any VirtualHosts, it's not nearly as flexible and means I can really only work on a single site at a time with some work necessary to switch back and forth between projects. Of course most of the time I just use Django's built-in development server when working with Django, but I do end up relying on Apache sometimes, and I'd like to set up my home server as a more complete development environment for both myself and some friends I can grant VPN access to. So to that end I've been looking into WebFaction's setup with the idea of re-implementing it myself.

Turns out it's pretty simple. Simple enough that I almost feel like I should have thought of it myself. Basically, WebFaction's setup scripts create a new 'app' in your ~/webapps/ directory, and populate it, most importantly with a copy, owned by your user, of the Apache executable, some scripts to start, stop, and restart that executable, and an httpd.conf file that sets the (in the case of a Python-based app) $PYTHONPATH variable to include a ~/webapps/yourappname/lib/python2.5/ directory allowing each site to maintain it's own dependencies independently (you can also put things in your ~/lib/python2.5/ for global dependencies if you want). Oh, each application also gets it's own copies of the necessary Apache modules to the same effect. Each application's Apache instance(s) is set up to listen to a different (non-80) port. The end result of this is an extremely simple, extremely modular setup that works fantastically.

Obviously I've left out a step here. If each Apache isntance is listening to a different, non-80, port, how does your traffic get to your actual site? This is the one part that I can't really just peek into the configuration files for, because it doesn't (as far as I can tell, which makes sense) live on the same server as my sites. I assume that what's happening is that WebFaction's name servers are simply pointing requests to (for example) joshourisman.com:80 at my.webfaction.server:portnumber. Again, a simple, yet elegant solution that allows for easy customization and expansion.

I haven't yet tried to implement this setup myself (I first want to move my server from FreeBSD to Linux (which now that I'm using full-time again I'm just much more familiar with), but there's nothing about it that's particularly tricky. Really, the routing is probably going to be the hardest part, but I'm planning on replacing our rather lackluster TrendNet wireless router with a Linux box which will give me much greater control and (hopefully) better reliability.

Django provides a lot of really useful tools to simplify the development process and let you focus only on the important bits. The FileField and ImageField (a subclass of FileField) are good examples of that letting you simply tell Django that your model will have a file or image and letting it take care of the issues of uploading, storing, and all that. In the past, that's really been enough. It will even automatically delete the file/image when you delete its parent model object. One thing it doesn't do, however (and this has been the topic of much debate), is delete a previously uploaded file/image when you upload a new one for the same model. What I mean by this is if you have a model with a FileField defined and use it to upload some file associated with an object. If you then later decide that you want a different file associated with that object and upload it, it doesn't delete the original file and instead leaves it in place and only changes the path stored in the database to point to the new file. Depending on the nature and traffic your website gets, this can lead to massive amounts of storage being wasted on orphaned files (assuming you don't want to keep those old files, of course). I toyed with a couple different approaches to this, including the possibility of subclassing the FileField to try and add that functionalty directly to the field. While this would probably work, I instead opted for a less generalized method: overriding the save() method of the model to take care of this:

def save(self, force_insert=False, force_update=False):
    try:
        old_obj = ModelName.objects.get(pk=self.pk)
        if old_obj.image.path != self.image.path:
                path = old_obj.image.path
                default_storage.delete(path)
    except:
        pass
    super(ModelName, self).save(force_insert, force_update)

This work perfectly, though it has the disadvantage of being specific to a particular model, which violates the DRY principle (assuming you're going to use it on more than one model). Fortunately there's a simple way to solve this problem too: subclassing models.Model and then instead of having all your models subclass models.Model directly, have them subclass your own version of it instead. In that case you'll probably want to have it work generically on all ImageFields and/or FileFields rather than having to name them all specifically. This isn't too hard, and you can build up a list of all the ImageFields for a particular model like this (taken from the sorl-thumnail project):

for field in model._meta.fields:
    if isinstance(field, models.ImageField):
        if field.upload_to.find("%") == -1:
            paths = paths.union((field.upload_to,))

[Edit: There was a bug in my code that I've corrected. Details are below in the comment by Comete.]
[Edit2: Fixed another problem in the code with variable names. Thanks, again Comete!]

Over to the right, in the sidebar, you may notice a nifty javascript calendar displaying the current month with some days highlighted. It is, in fact, a navigation tool for the blog. The highlighted days are the days with posts. Click on a day, see the posts for that day. If there are no posts for that day, it falls back to the month. If there are no posts for that month, it falls back to the year. Pretty simple stuff, really.

The calendar is implemented with YUI, and the fallback mechanism is, of course, all Django code. The fallback isn't something that's built into Django, sadly, but was relatively easy to implement by just wrapping Django's date-based generic views in some views of my own. Really it's just a simple 'try except' block that catched a 404 error and instead returns a redirect. Not too shabby, if I do say so myself:

from django.template import loader
from django.http import Http404, HttpResponseRedirect
from django.views.generic import date_based


def archive_month(request, year, month, queryset, date_field, month_format='%m', template_name=None, template_loader=loader, extra_context=None, allow_empty=False, context_processor=None, template_object_name='object', mimetype=None, allow_future=False):
    try:
        return date_based.archive_month(request, year, month, queryset, date_field, month_format, template_name, template_loader, extra_context, allow_empty, context_processor, template_object_name, mimetype, allow_future)
    except Http404:
        return HttpResponseRedirect("/%s/" % year)

def archive_day(request, year, month, day, queryset, date_field, month_format='%m', day_format='%d', template_name=None, template_loader=loader,extra_context=None, allow_empty=False, context_processor=None, template_object_name='object', mimetype=None, allow_future=False):
    try:
        return date_based.archive_day(request, year, month, day, queryset, date_field, month_format, day_format, template_name, template_loader, extra_context, allow_empty, context_processor, template_object_name, mimetype, allow_future)
    except Http404:
        return HttpResponseRedirect("/%s/%s/" % (year, month))

(Also check out nifty syntax highlighting! Based off this snippet from Django snippets, modified to remove the Markdown bits. And by highlighting, I of course mean formatting. I'll be adding fun colors and such to it in the near future.)

I've just finished implementing comment moderation for the blog using Jannis Leidel's fork of James Bennett's django-comment-utils. I've run a few test which seem to indicate that the basic comment moderation functionality is working though, at the moment, I still don't have notification emails going. It was actually rather amazing just how much comment spam this blog got in the past two weeks since it went live on Django without spam protection. But this should significantly cut back on it. It's really great having such a huge body of quality work out there to draw from when building a Django project.

[Edit: Looks like the problem I'm having with email may be on WebFaction's end rather than mine. Nice because it means I probably am doing things right; annoying because it means it's not within my power to fix it. Oh well, a ticket has been submitted and hopefully soon I should be getting email notifications of new comments.]

I just submitted my first patch to Django! Among other things, this is my first real forray into the inner depths of the Django code. This patch fixes an issues that had been bothering me for quite some time. In Django's admin interface it's possible to specify that a particular field should be automatically filled in with the value(s) you enter in some other field(s). For example, as I typed 'My first Django patch' into the title field of the form I used to write this post, it was automatically filling in a slug field that's being used for the permalink to this post with 'my-first-django-patch'. This is a very useful features and uses just a little bit of javascript to accomplish it. The only problem is that it only works when you're trying to pull information from a text field. Sometimes, however, you might want to pull information from another sort of field, such as a drop-down menu. Previously Django simply wasn't capable of this. With my changes, however, it is able to handle this potentiality quite well.

It's not really a huge patch, just a fairly a little added code to a single javascript method, and there's no guarantee that my patch will every make it's way into the Django code base, but it's still fun to be able to contribute to one of my favorite open source projects ever. The patch, for those that are curious, can be found here, and I've also submitted it to djangosnippets.org here.

I've just implemented the first real benefit of having my blog now be Django based instead of WordPress based. Because part of the reason I had wanted to make this move all along was to allow for some close integration between my blog and my business site, when I went about setting up the Django project for the blog I actually just duplicated the project for my business site. Because of this, I'm able to use Django's sites framework to have both project pull from the same database. This means that any information available to one is also available to the other.

Because I used a ManyToManyField to define the relationship between a blog entry and a site, I'm able to specify that a particular entry is related to either or both of the two sites (as well as any future sites that I may decide to add). This entry, for example, is related to both. Thanks to Django's fabulous templating system I was able to effortlessly integrate the blog templates into my business site without every writing a single line of HTML or CSS.

The end result of all this? The business relevant posts on my blog are available not only at joshourisman.com, but now also at dydxtech.com/blog.

How cool is that?

Hot on the heels of having imported the old posts from the defunct WordPress version of my blog, I've now imported the comments as well. Unfortunately this proved a lot trickier than the posts, and many comments very likely did not make it over. Just over 200 did, however, and, for now, I'm going to call that good enough and focus on other things. Any comments left from now on, however, will work just fine thanks to the awesomeness of Django, and it's comments framework which is so incredibly simple to use I don't think it's worthwhile to do any more than simply link to the docs.

It's always fun doing this: a new project has gone live! In cooperation with BostonChefs.com, Fresh from the Family Farm is an event for local Boston-area restaurants to showcase the possibilities of local ingredients from area family farms. The event runs from Oct. 12 through Oct. 19, and a list of participating restaurants, along with links to make reservations through OpenTable (where available), and links to the restaurant's BostonChefs.com profile (also where available).

The website is powered by Django (which is now at 1.0, something I plan to write about soon), which even still I am growing to love more and more as I use it.

Since last night when I posted the last entry I've added a few things to my new blog. First off, syndication. Anyone who was previously subscribed to my blog through the FeedBurner feed (http://feeds.feedburner.com/joshourisman/lRQx), which should be anyone who was subscribed to it, should hopefully still be subscribed. If not, please let me know either through a comment on this post or via email.

I've also added XMLRPC pinging to it, so every entry I post should be submited via Ping-o-Matic to all the normal places.

Probably less importantly, I've added a Google site search box to my template so you can search my blog, although at the moment it's mostly going to return results from the old blog which are currently unaccessible (but I'm working on that).

Next steps, other than getting my old content imported into the new blog, are Akismet spam filtering for comments, better integration of tagging (which is actually already there, just not in a usable manner), post archives (which are pointless until I get that data imported), and better templates.

As this is all being built entirely from scratch feedback is, of course, both encouraged and welcomed.

As you may recall, I've mentioned a fair number of times in the past that I've been planning to migrate my blog away from WordPress on onto my own custom Django-based solution. Part of the reason for wanting to do this is fun, but also because I'd like to be able to have a little more integration between my blog and my business site.

If you're reading this, you can no doubt tell that something isn't quite right with my blog. Mainly, it looks completely different, this may well be the only post you can see, and it might be lacking a feature or two that it had before. The reason for this is that my hand has been forced. For the past several weeks I've been unable to log into my blog. No, I didn't forget my password, it just won't let me log in (it even says I have the wrong password when I try something else). This simply will not stand.

So I've thrown together a very very quick and barebones blogging app (and when I mean quick, I mean I did this in less than two hours) that at least gives me some ability to keep blogging. I will, of course, be expanding it to add the missing functionality, and I will be working on importing the posts, comments, and such from the old blog (I still have access to the databases, so this shouldn't be a problem). In the mean time I'm planning to set up some redirects so that links to old posts aren't broken.

Hopefully it won't take very long for me to get this new blog up to speed in terms of functionality and design, and to get my old stuff imported into it so I can finally be rid of WordPress entirely. Of course, I do have to prioritize projects for clients, so it may take a bit longer than I'd like.

I've just started working on a new project using ExpressionEngine, a PHP-based content management system. I didn't really know a whole lot about it going in (it was the client's choice to use EE), but from what I've seen so far it's a pretty decent piece of software.

Being a CMS, it sits somewhere between blogging software like WordPress, and a framework like Django. Basically this means that it's much more structured than Django, proving a lot more blogging and related functionality out of the box but still much more flexible than WordPress allowing for a lot more freedom in the creation of your website. So far it seems like they've struck a good balance, providing a relatively shallow learning curve while still giving you a lot of powerful features.

Of course, for me, I found the more structured nature of it to be a bit restrictive. Also, I just don't get some of the choices they made. For example, your URLs, while readable and search engine friendly, all look like 'domain.tld/index.php/blog/archive/&c;'. Not a bad URL, but why on Earth is that 'index.php' in there? If they're going to use URL re-writing to provide nice URLs, why do they leave that useless bit of information in there? It's not like it would have been any harder for them to have taken it out. Also, the only obvious way to edit templates is through their web-based control panel. I'm sure there's really nothing stopping me from going in with an FTP client or via SSH and editing the text files directly, but they don't even hint at where you would want to look to do that (and as I didn't install it on the server myself I have no experience with the directly structure). Neither of these problems is a big deal, or a big obstacle to someone who knows what they're doing and wants to change it, but they just seem like very strange design decisions to me.

Overall, however, I think it looks like a pretty good system, and very good way to rapidly build a flexible and highly useful CMS. I don't know that it'll ever be my first choice for a project, as anything I can do with EE I can also do with Django, and in a way that's more intuitive (though probably has a higher initial investment of time), and anything that I don't need that level of flexibility and power for, I'd probably just use WordPress. But still, it's a nice piece of software, and I can definitely see how it would be a very good choice for a lot of people.

As those in the Boston area are probably aware, this years Summer Restaurant Week is fast approaching. This year it will be two weeks, those of August 10 through August 15 and August 17 though August 22. If you've been reading my blog for a while you may know that one year ago, almost to the day (off by 5) I announced that I had helped work on BostonChefs.com's Unofficial Guide to Restaurant Week. Well, this year I'm announcing the same thing.

The site just went live with all the information you might want about what's going on with this Summer's Restaurant Week including the restaurants that are participating, what meals they'll be serving, what's on their menus (where available), and a Google Maps mashup to help you find them. The site has been completely redeveloped and is now powered by Django, making this my first Django-based project to go live! (Not counting my own website, of course.)

Go check it out, and bon appetit!

As I mentioned in my last post, I recently migraded my dy/dx tech website to a different hosting company. If you've really been paying attention, you may recall that not too long ago I had gotten a Media Temple hosting account with the plans on migrating all of the sites I host, both my own and clients' to it only to discover that setting up Django on a Media Temple (dv) account is far more trouble than it's worth. My estimation of that hasn't changed, in fact I actually cancelled my Media Temple account a few weeks ago after the last client I had hosted there was moved off to another host. My experiences with WebFaction have been so positive (exploding data centers notwithstanding), that I have instead migrated everything to their servers. Well, not everything yet. This blog is still hosted on DreamHost for the time being (though I plan on moving it to a WebFaction hosted WordPress blog in the very near future before eventually migrating it to a Django based solution as I've mentioned before).

The hosting hassles referred to in the title, thankfully, have nothing to do with the actual hosting companies I'm dealing with, and are instead due to a foolish mistake on my part: when I switched my domain to WebFaction, I forgot that I had custom MX records enabling the use of my hosted google apps for my domain. As a result, as the new DNS information started propagating, people stopped being able to send me email. Fortunatly, it was an easy fix to just change the MX records with WebFaction, and I don't think I missed any important emails, but if anyone out there got a bounceback when sending me an email, that's why.

It's been a while since I've been able to announce a big new project. Not because I haven't had any, but because everything I've been working on lately has been so large that nothing is quite ready to go live yet. But finally, I get to announce a big project that I recently finished: the Becoming MOBOS video blog. As I'm sure many of you from the Boston area are aware, there is a new Mandarin Oriental that's been under construction down by the Pru. They hired me to create an internal video blog for them. Unfortunately, since it's internal, I can't link to it, but the screenshot to the right links to a full-size, albeit redacted, image. It's a WordPress based blog using a verstion of WPelements.com's MassiveNews theme customized by your truly. I also used FlowPlayer to provide the Flash video playback capabilities. All in all, I think it turned out to be a pretty slick site.

That's not the only news, however. In preparation for announcing the Becoming MOBOS site I've been doing a little work sprucing up my own website. So I also get to announce a new version of the dy/dx tech website (I also changed hosts for it, so you may need to wait for the DNS to propagate if you're still seeing the old site). The overall look of the site is the same as before, but I've removed some rather pointless elements such as the Google Map that used to be on the front page. In it's place is now a slideshow of screenshots from my portfolio, which I think is a much better use of the space. The majority of the changes, however, are under the hood. As you may recall, I redeveloped the site using Django a while ago. Since then I've spent a lot more time with Django and know a lot more about it, so I completely redeveloped the site (using the newforms-admin branch and was able to make a lot of improvements to the code, and basically leave it better positioned to integrate more features in the future. Among other things, I plan on migrating this blog to a Django-based solution and integrating it into the dy/dx tech website to some extent. I've been working heavily with Django for the past several months, and I just keep liking it more and more. It makes every part of my job so much more enjoyable and, in a lot of cases, faster. Be on the lookout for another project going live in the next couple weeks: this one will be Django-based and will be very public, and, I predict, very popular.

Currently I've got two projects hosted on WebFaction servers. So far, I really like them. As managed hosts go, they're probably the best I've worked with, and they certainly make life very easy when building Django powered sites.

Today I got an email from one of the clients whose project is hosted on WebFaction saying that their site is down. So I checked it out, and while I was able to access it, it was extremely slow, to the point where a less forgiving browser/LAN setup might cause it to time out. So I fired off a support ticket to WebFaction, and within a couple minutes, not only was the site back up to speed, but I was provided with a very good explanation for why my server was having problems.

Apparently there was an explosion at one of WebFaction's data centers this weekend. It took out power to the data center, but fortunately no one was hurt and none of the servers were damaged. Obviously, there have been some interruptions in service for the servers in that data center (which includes both of my WebFaction projects), but they've already gotten a significant number of the servers back online (though only one of mine).

Amazingly, this is actually the second time I've had a server taken out by an explosion at a data center. The first time was with a hosted Microsoft Exchange server with a hosting company in London.

It really sucks having sites down, especially critical ones (fortunately only one of the projects I have hosted with them is critical, and it's the one that's back up already), but as reasons for downtime go, you have to admit that an explosion is a pretty good one.

I'm currently working on a fairly large Django project that I think I've mentioned a couple times in the past. In that Django project there is a Person model and an Organization model. Both Persons and Organizations have email addresses. Organizations are related to Sites (through a ManyToMany field), but Persons are not (they're related to Organizations through an intermediary table). I'm currently attempting to create a contact form, so that people can email either a Person or an Organization using newforms.

For security reasons, we don't want the user to actually see the email address, just the name of the Organization or Person. Fortunately, newforms has the ModelChoiceField class that you pass a queryset of options. In keeping with DRY principles, I want to be able to use a single Form regardless of whether the person is trying to email a Person or an Organization (both models have a field named 'email_address'). Unfortunately, this is where I ran into my first problem: ModelChoiceField doesn't really allow you to define the queryset dynamically, you have to define it in the form definition. Luckily I found this blog entry which provides a method to re-define the queryset in the __init__() method which allows you to change it based on the HttpRequest object. My next problem was that I only want the user to be able to email Organizations and Persons on the current Site. Since Organizations are directly related to Sites I just used the CurrentSiteManager. However since People are not directly related to Sites, and are instead related only to Organizations (through an intermediary table), I couldn't do this. Instead, I decided to try this crazy bit of code:

Person.objects.filter(persontoorganization_map__organization__in=Organization.on_site.all())

Amazingly, it just worked exactly as I would have wanted it to. No fuss, no problems, just a queryset of Persons related to the current site. Go Django!

As you may recall, a while ago I got myself an account at MediaTemple with the idea that I'd move all my websites over to there. I had previously been using Dreamhost, but wanted something a little more high quality so that I could reasonably offer hosting services to some of my clients. MediaTemple seemed like a good way to go, and for the most part their service has been great.

Unfortunately, I have run into a few problems. Most importantly, despite spending a fairly significant number of man-hours working on it, I've been unable to get Django running on my (dv) server. Yes, they have a (beta) program that makes it easy to run Django on a (gs) account, but for my needs a (gs) simply won't do and I really don't want to have multiple accounts with them. The end result of this is that several of my web pages are still running on Dreamhost because they require Django (this blog actually is as well even though it's currently a WordPress blog, because I want to switch to something Django-based and it seems like an unreasonable hassle to migrate my WordPress blog to a new server only to then have to migrate it again to new software, especially as I'm currently holding onto my Dreamhost account for my Django-based pages anyway).

The issue is now coming to a bit of a breaking point. Why? Because I'm currently working on a pretty large Django-based website that will be going live in the next month or two. For the purposes of development, it's being hosted on WebFaction, which has been an amazing host. They make it incredibly simple to host a Django site, to the point that basically zero setup is required. But as we get closer to the point of going live, I've been considering what the hosting needs of the site will be going further, and how to best serve them.

The site is a redevelopment of an existing site, so we can get a pretty good idea of what the traffic numbers are going to look like. This will let us extrapolate the RAM and bandwidth requirements pretty well too. The issue, is that once the Django version of the site goes live, we're going to start to expand it. Thanks to the capabilities of Django, it's being developed with the potential for massive growth in mind. Specifically, it's using Django's Sites framework to allow for expansion to several sites. Currently there are only two, but the Django version will go live with 4 or 5, and there's the potential to expand far beyond that.

This means that we're going to require a pretty large number of (software) servers. There's the MySQL server running the back-end, an HTTP server for static content, and then an HTTP server for each site, and they're all going to be using up resources to different extents. Trying to find the best hosting solution for this sort of setup has led me to a couple of options:

1. Stay with WebFaction. A number of people have said they believe that WebFaction's shared hosting plans should be able to accomodate this. WebFaction provides a very good combination of ease of use and low-level access, their prices are good, and the way they have their hosting set up, it's extremely simple to add another Django install complete with its own Apache/mod_python instance. They also offer dedicated server, but I think there are probably better routes to go than with WebFaction's dedicated servers.
2. Switch to Slicehost. I've only just learned about Slicehost, but so far they look like a pretty sweet deal. For a very reasonable price you get a virtualized server running a Linux distro of your choice (you can choose from 8 right now) run on Xen. They claim not to oversell their servers, so you're guaranteed to actually get the full capacity that you pay for (unlike with budget hosts such as Dreamhost). And since you're getting your own virtualized host you have full root access. They basically have nothing preinstalled, so you can easily set it up in whatever configuration you want without having to deal with the vagaries of the anointed hosting package (Plesk on MediaTemple, I'm looking at you). I really like like look of them, and they've been getting good reviews. They're currently listed as the number 2 hosting company on Djangofriendly, behind only WebFaction. With my background in IT and Linux administration, the fact that I'd have to manage everything myself isn't enough to scare me away either. The fact that they let you choose your Linux distro really appeals to me as well, as I'm by far more familiar and comfortable with Gentoo than any other Linux distro. I only wish they offered FreeBSD slices, but the only reason they don't is technical, and once that issue is resolved it sounds like they plan on it. In a lot of ways, they're basically a very affordable colocation provider. The biggest issue, it sounds like, is that apparently communication between different slices counts against your bandwidth allotment (for both slices, presumably). This means that as the site I'm working on grows, if it spreads out to multiple slices (which it undoubtedly would and which I'd want it to do since that will give the added reliability of spreading across multiple physical machines) we'll basically be billed for database access from the sites that aren't on the slice with the database server.
3. Colocation. Colocation is basically the 800 lbs. gorilla in the room. It costs a lot more, but you get what you pay for. With colocation we'd have all the advantages of Slicehost (minus the low price, of course) plus the ability to expand more or less arbitrarily. We could have as many physical machines as we wanted running as much or as little of the site as we wanted. Provided we're willing to pay, of course. On top of that all the server management would again fall to me, but this time without some of the nice shortcuts that Slicehost offers. Essentially, colocation is alway the fall-back option. But hopefully one of the other two hosts can offer us a solution that's a little more balanced: we get less control, but more simplicity and ease of use for a greatly reduced price.

At the moment, I'm leaning towards sticking with WebFaction for now. I'm already very impressed with what they offer, and from the sound of things, they'll continue to be a more than adequate host as we expand. But I'm also definitely looking for input. If anyone has any suggestions or recommendations I'd love to hear them. In particular, any first-hand experience with hosting large Django sites with any of these solutions are most welcome.

So, a little while ago I got a MediaTemple (dv) server. In general, they seem to be pretty good. The server's been fast and reliable so far, I get a good amount of storage and bandwidth for the price, and I have root access so theoretically I can do pretty much whatever I want. I have run into a bit of a problem however.

For the past several days I've been banging my head against installing Django on my MediaTemple server. In theory this should be pretty easy. The server comes with Apache and mod_python installed, so all that I should need to do is check out the lated django trunk from svn, make a few symlinks, install MySQLdb, change some Apache configurations to tell it to use Django for the appropriate URLs, and go. In practice, it's not working quite so well.

The first few steps were simplicity itself. It wasn't long before I had a successful Django install and was able to 'python manage.py syncdb' to have my projects database tables created. Changing the Apache settings was a little more difficult because you can't actually modify the httpd.conf as Plesk will overwrite it. Of course, there's no documentation telling you this or telling you the appropriate solution (despite, I might add, the insistence of the MediaTemple KnowledgeBase that instructions for installing Django on a (dv) server exist somewhere). A little help from Django's Google Groups community helped me out with that, though, letting me know that I instead need to create a /var/www/vhosts/MYDOMAIN.COM/conf/vhosts.conf file and put the settings in there. So I did that, and it even seems to work, but when I try to actually go to the page I get this:

Mod_python error: "PythonHandler django.core.handlers.modpython"


Traceback (most recent call last):

  File "/usr/lib/python2.3/site-packages/mod_python/apache.py", line 299, in HandlerDispatch
    result = object(req)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/modpython.py", line 188, in handler
    return ModPythonHandler()(req)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/modpython.py", line 161, in __call__
    response = self.get_response(request)

  File "/usr/lib/python2.3/site-packages/django/core/handlers/base.py", line 64, in get_response
    response = middleware_method(request)

  File "/usr/lib/python2.3/site-packages/django/contrib/sessions/middleware.py", line 15, in process_request
    request.session = engine.SessionStore(session_key)

AttributeError: 'module' object has no attribute 'SessionStore'

Apparently, for some reason, there's some problem with Django's sessions middleware, although no one else seems to have discovered this problem. If I remove sessions (and therefore the admin app) from my project the page will actually load, but it then fails to actually be able to get anything from the database so gives me errors on any page that requires database calls (most of them). Thus far neither the Google Groups nor the expert aid of Jesse Legg have been able to help me make any progress in solving this issue.

There are people out there running Django on MediaTemple (dv) servers, so I know this is possible, and I'm confident that I'll get it eventually. But in the mean time it's incredibly frustrating and is putting serious delays in the process of transitioning my sites over to MediaTemple (not to mention in the development of some other Django projects I'm working on right now). You can be sure that when I finally get Django working on my (dv) server that I'll be writing a detailed account of exactly how it was done. There really needs to be some documentation on this out there.

I've been meaning to do it for a while, but I've finally gotten around to creating a portfolio. Since my website is now Django based this was incredibly easy, and probably only took about 45 minutes to do. Previously I've just been keeping a list of links to former projects that I included in emails to prospective clients. I didn't really want to put up a portfolio when I only had a handful of projects to show off. But in the past few days three different projects went live: Lola Boston for which I created the locations database, Tundratour for which I created a database for the different trips as well as a 'shopping cart' to allow people to request more information on multiple trips at once, and Sel De La Terre which was live before but now has a tool for purchasing gift cards online that I created. I figured that 5 projects was a big enough number to go live with, especially as it should be growing pretty rapidly in the next few months.

I'm still trying to decide if I want to put more information in there. Should I put in a little description of each project, or at least expand on what my contribution to the project was? The Chainsaw Awards page was nominated for the MITX awards, I should probably mention that somewhere. Fortunately, now that it's there it will be easy enough to add more to it. The other question I've been asking myself is whether or not I should include websites that I worked on at my old job. I was just as much responsible for those projects as the ones I'm doing now, but somehow it just seems like I should leave them off. Fortunately again, it will be easy to add those later if I decide to. And in the meantime I have a portfolio to show off. A pretty nice one, if I do say so myself.

Also, I do still intend to write that post that I promised while I was in Jamaica, I've just been very busy ever since getting back. I'm going to be away again this weekend (Florida for another wedding), so hopefully I'll be able to get to it next week when I'm back.

My first Django project is now live. Sadly it's not a very interesting one, just a re-develop of the dy/dx tech website. It looks exactly the same as before, but it's now powered by Django. This doesn't really offer any advantages at the moment, but it will. For example, pretty much all the data on the site is currently stored in a database which means it will be extremely simple to add, remove, or change any of the services show in the services tab. Not that I really expect that to change any time soon (although you may notice that the services tab is the one part of the website that has changed; I've added a few, and consolidated some redundant ones), but the principle is sound.

More importantly having the site powered by Django will make it much easier for me to add some new features/online services that I've been thinking about for a while. The first one will definitely be a portfolio tab. I've worked on a pretty good number of websites in the year or so since I started this business, and I really should have a portfolio on my website to show off my work. I'd also like to put up a clients tab where I can list my clients and, if I'm lucky, get some testimonials to put up there as well. Then of course there's the WiFi database that I've been talking about for a while, that will have it's own subdomain, but I'll give it its own tab as well. Hopefully now that the whole site is done with Django and I'm a little more familiar with how the framework works development of those things and others will go a little faster.