Django and database migrations using South

Currently I am working on a Django project that uses multiple databases. To migrate the databases automatically when the asscoiated models change, we use South. Unfortunately South cannot handle ForeignKeys between models from different databases, see this thread on the South Users Google Group. When South tries to execute the migration step that introduces such a relationship, it aborts the migration and leaves me with an incomplete database. This also causes a problem when running unit tests as South's test runner integration will try to build the database and fail. Consequently, no tests are run.

It is very easy to work around this:

1. Disable South's test runner integration: set the option SOUTH_TESTS_MIGRATE in settings.py to False


2. Disable South in you Django project: remove 'south' from the tuple of installed apps (in settings.py)


3. Create the database using Django's syncdb: bin/django syncdb


4. Enable South: add 'south' to the tuple of installed apps (in settings.py)


5. Fake the migration up to the last step: bin/django migrate <app> --fake

South uses a special table in your database to register which migrations have been performed, south_migrationhistory. After the last step this tables contains all the steps.

Debugging Python code from within Emacs on Windows

It is very easy to debug Python code from within Emacs. However, it does not work as advertised on my Windows XP development machine - I do not know whether it works as advertised on Linux. The problem is that the Emacs buffer to interact with the Python debugger (Pdb) only displays the directory of the source file I want to debug:

Current directory is c:/projects/customer X/scripts/converter/converter

In case the problem is related to the Emacs version, I am using version 23.1.1.

After some investigations, I found out that there two things I had to take care of to get the interaction with Pdb working.

1. To start Pdb from within Emacs, you M-x the command "pdb" (without the quotes). You then have to enter the command-line required to start Pdb. On my Windows XP machine that is

python c:/Python26/Lib/pdb.py updateweights_tests.py

Emacs parses the output of Pdb to stdout to let the user interact with Pdb. By default, this output is buffered on Windows and Emacs can only parse the part of the output that has been flushed. This can result in a deadlock where Emacs is waiting for more PDb output and Pdb for user input.

To turn off buffering, you have to supply the Python interpreter with the command-line option -u:

-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'

The command-line to start Pdb becomes

python -u c:/Python26/Lib/pdb.py updateweights_tests.py

2. Function gud-pdb-marker-filter (gud.el) uses regular expression gud-pdb-marker-regexp to identify specific parts of the Pdb output. This regular expression is defined as follows:

(defvar gud-pdb-marker-regexp
"^> \\([-a-zA-Z0-9_/.:\\]*\\|\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\|\\)()\\(->[^\n]*\\)?\n")

The first group of the regexp should match any file path. However, it fails to recognize paths that contain a space, such as the path that contained my sources. This was easily fixed by the addition of a space to the first group:

(defvar gud-pdb-marker-regexp
"^> \\([-a-zA-Z0-9_ /.:\\]*\\|\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\|\\)()\\(->[^\n]*\\)?\n")

I use nosetests to automatically collect my unit tests and execute them. It is very easy to run nosetests from Emacs and automatically invoke Pdb as soon as a unittest.TestCase assertion fails. First, create a Python module that invokes nose.run():

import nose; nose.run()

Then M-x the command pdb and enter the following command-line in the minibuffer:

python -u nosetests.py --pdb-failures --nocapture --quiet

The options "--pdb-failures --nocapture --quiet" are intended for and automatically picked up by the call to Nose.run. The nosetests "Usage" message has the following to say about these options:

--pdb-failures Drop into debugger on failures
-s, --nocapture Don't capture stdout (any stdout output will be
printed immediately) [NOSE_NOCAPTURE]
-q, --quiet Be less verbose

The "--quiet" option is required so nose does not output messages that make it impossible for Emacs to parse the Pdb output.