'import site' failed; use -v for traceback

If you suddenly start seeing this error every time you try to run Python:

'import site' failed; use -v for traceback

Chances are you have an erroneous file in your site-packages directory that's screwing up Python's import path initialisation and could lead to you being unable to import some installed modules.

To figure out which file this is you can use the Python debugger, pdb:

> python -v
>>> import pdb
>>> pdb.pm()
> [python_lib_path]/posixpath.py(173)exists()
-> return False
(Pdb) u
> [python_lib_path]site.py(146)addpackage()
-> f.close()
(Pdb) f
<closed file '[site-packages]._blah.pth', mode 'rU' at 0x52c38>

Your actual output will be quite a bit different with a lot of debugging information spewed into the console but let's explain what's going on here:

1. Run Python in verbose mode to get a traceback.
2. import pdb
3. Run the pm (postmortem) function to enter the (Pdb) shell and inspect the exact point at which the error occured.
4. Go up a level in the stack with u.
5. Inspect the f file object.

This tells us that ._blah.pth is causing Python's site import routine to choke. [bug report]

Move this file out of site-packages and try again, repeating the steps above if you get the same error.

When you run Python, it runs through all the directories specified by your PYTHONPATH environment variable, looking for .pth files. These can contain further directories to be added to the import path and are typically created by python packaging systems like easy_install. [more info on .pth files]

So what's up with this one?

The so called "dot-underscore" or AppleDouble files are actually created by Mac OS X. They contain resource forks and extra file metada for their corresponding file (the example we found earlier contains metadata for a legitimate file blah.pth).

Mac filesystems like HFS+ support resource forks natively and don't need these files, so they aren't usually created when you're accessing your machine locally. They can get created when you're accessing it over a network using the Samba protocol or if the connecting Mac otherwise thinks you don't support resource forks—usually if you're on Windows or an NFS filesystem.

You can safely delete these files from the site-packages directory and Python should be back to normal.

Safari 3.2.1 bug: Random unicode characters munged into select drop down

I recently encountered a very bizarre browser bug involving a <select> drop down in Safari 3.2.1.

Changing the selection was resulting in random characters overwriting the text contents of the selected <option> element.

Since this behaviour wasn't happening in the latest version of WebKit, my bug report was resolved WORKSFORME.

This wasn't particularly helpful as I still needed to figure out a workaround for a bug that was affecting people using the current shipping version of Safari.

I eventually managed to isolate a minimal test case here:
http://james.wheare.org/stuff/bugs/safari/fixedselect

The setup looks like this:

<div id="relative" style="position: relative;">
    <div id="fixed" style="position: fixed; top: 0; left: 0;">
        <select>
            <option>Choose another option</option>
            <option>Garbled</option>
        </select>
    </div>
</div>

The bug is triggered by javascript:

// Set the fixed container's positioning to absolute
var element = document.getElementById('fixed');
element.style.position = 'absolute';
// Access its clientWidth or clientHeight properties
element.clientWidth;
element.clientHeight;
// Restore the positioning to its original value of fixed
element.style.position = 'fixed';

Now, just select a different option from the drop down and it's text content will be munged with random characters.

This may seem like quite a strange sequence of operations but it's actually used in the Prototype library's Element.getDimensions method [source].

When I discovered this bug, it was additionally causing Safari to crash if you continued to use the drop down. This test case doesn't trigger the crash, but it's almost certainly the same root cause.

There a few options to workaround this issue, depending on your particular situation:

1. Avoid Element.getDimensions.
2. Modify Prototype's Element.getDimensions function.
3. Change the positioning of your containers.
4. Wrap the select in another position: fixed element and don't call Element.getDimensions on it.
5. Wait for Apple to release a new version of Safari.

If you choose to modify Element.getDimensions, there's a handy way to do that without altering the Prototype source code. Simply define your own version and then extend Element with it by using Element.addMethods. This will override the built in method.

Element.addMethods({
    getDimensions: function(element) {
        // Your version in here
        // ...
    }
});

If you're wondering how to patch Element.getDimensions, I forked Prototype on github with a fix for this issue:
Fix for Element.getDimensions

I also filed a ticket for Prototype to add a fix for this themselves:
Element.getDimensions causes issues in certain conditions in Safari
Update: This fix has been accepted into Prototype and should make its way into a future release.

Ordering related objects in Django

Django models have a meta option order_with_respect_to that allows you to order objects within the scope of a related ForeignKey object.

This option adds an _order column to the model's database table to keep track of this ordering.

That's all very well, but if you need to change the sequence order you may well feel at a loss. The official documentation makes no mention of this option beyond a basic explanation of its purpose , and editing your objects in the built in admin app reveals no user interface for changing the ordering either. [1]

Update Feb 2009: There's now a ticket for this functionality to be added to the docs.

I managed to uncover a Python interface by delving into Django's API internals [2]. An object with related models that order with respect to it is given two handy methods:

get_RELATED_order()
set_RELATED_order()

where RELATED is the lowercased model name of the ordered objects

class Entry(model.Model):
    # ... fields

class Comment(model.Model):
    entry = model.ForeignKey(Entry, related_name='comments')
    # ... more fields
    
    class Meta:
        order_with_respect_to = 'entry'

Given the above example models, you can retrieve the order of an entry's comments:

>>> entry = Entry.objects.latest()
>>> entry.get_comment_order()
[1, 2, 3, 4]

And change the ordering by passing a list of comment ids back in.

>>> entry.set_comment_order([2, 1, 4, 3])

N.B. Be sure to pass in the same ids returned by get_comment_order

Two other handy methods exist for the Comment objects, get_next_in_order and get_previous_in_order [3]

>>> comment = Comment.objects.get(pk=3)
>>> comment.get_next_in_order()
<Comment: 4>
>>> comment.get_previous_in_order()
<Comment: 2>

1. A very early ticket is still open to restore this functionality to the admin app, and recent activity suggests it may well reappear soon. See also this thread on the django-developers mailing list. back


2. Look particularly at the two helper functions method_set_order and method_get_order in django.db.models.base. back


3. Theoretically at least, there's still a bug in get_previous_in_order but it should be fixed soon. Update: Fixed by the queryset-refactor back

Barcelona 11/07 Photos

Bus times back online

LiveBus.org is currently the first hit for "mash up offline" on Google. Not a particularly delightful situation, but I'm happy to say that the DeadBus has risen!

I've moved the site to a sultry new virtual private server at RimuHosting and real time bus updates are flowing once more. You shouldn't notice a difference, but I've now got a lot more freedom to tweak performance behind the scenes, and future downtime of this sort should be rare.
Disclaimer: Rare, not impossible

Do let me know if you're seeing funny stuff, things may have become muddled in the move.

LiveBus.org down

Due to some server configuration issues, I've had to take LiveBus down until further notice. I'll hopefully have the problems solved soon and will make another announcement as soon as the site returns.

Sorry for the annoyance.

Update: The site's back online

LiveBus.org launched

As part of a demo at the inaugural Oxford Geek Night (Update: podcasts now online at that link - hooray for immortalised mumblings!) on Wednesday night, I finally gave in and launched the project I've been working on for the last many months.

LiveBus.org is a free service that lets you find bus stops and routes on a map and look up live arrival times. It currently covers most routes in Oxfordshire and Surrey, and will hopefully be expanded in the future.

More info on LiveBus is at the about page.

Comments/questions/suggestions welcomed and encouraged.