Entries Tagged 'django' ↓

I and a host of other folks have been waiting anxiously for this merge for a while now. Excited to see it happen tonight.

Good job, Brian Rosner et al.

I have been desiring to have the documentation that I write for my various projects on GitHub render to HTML when using a .markdown extension, just like they support doing for special files like README.markdown.

At first, I thought they’d eventually get around to it and that is the best I could hope for as surely they are too busy to respond to a non-paying customer like me. Then I learned they the used the Lighthouse ticket system to manage their project and took public submissions for features.

Before logging a feature request I did some searching and found a ticket already out there requesting exactly this same feature that I wanted. So I just added my own comments to the ticket to give it a +1.

After a little back and forth I learned it was implemented tonight (just a week after my first comment). Good to see such responsive development.

An example of this feature can be found in my django-aws project which is simple app to provide some template tags to provide Amazon Web Services information a la boto.

In my continued experimentation with the newforms-admin branch of Django, I wanted to figure out how to order fields of an inline. Looking at the documentation for inlines I saw there was not an ordering field.

I had thought that there was but it turns out I was just mistaken in the object hierarchy. InlineModelAdmin inherits from BaseModelAdmin the same as ModelAdmin — I was thinking that InlineModelAdmin inherited from ModelAdmin.

Therefore, I had to determine a way to accomplish this via some spelunking through the code. At first, I thought I’d just create my own template by inheriting or copying the tabular.html template. After looking at it, I determined that I didn’t want to figure out how to do a regroup on inlineadminformset or if even that was the proper way to try to order things in that template.

After a few minutes in the code I realized that I could just subclass BaseInlineFormset and specify the fields that I wanted to order by (in this example I will use starttime and endtime:

from django.contrib import admin
from django.newforms.models import BaseInlineFormset

class MyOrderedFormset(BaseInlineFormset):
    def get_queryset(self):
        qs = super(SessionInlineFormset, self).get_queryset()
        return qs.order_by('start_time', 'end_time')

class MyOrderedInline(admin.TabularInline):
    model = MyModel
    extra = 1
    formset = MyOrderedFormset

That’s it. Now after that section of inlines are ordered by starttime and endtime.

Introduction

Managing database changes in a team environment working on a django project can be complicated. I would imagine that there is no one size fits all solution and it would depend on team size and configuration, production database size, etc.

What I will outline here is a solution I developed for a small team that I work with on a django based web application. So far it has worked really well for allowing us to streamline database changes so we can stay in sync, deploy easily, and add small tweaks to the database (indexing, data manipulation, etc.) that is semi-automated.

The idea is based partly on Rails database migrations (though not has sleek and well put together) and a database versioning system that was used on a team that I have worked on previously using MS SQL in a corporate environment and a fairly decent team size. In one sense it is not nearly as well put together as either of these two solutions, but at the same time, it is pretty much hands off and has been working well for us for a number of months now.

Why not just use syncdb?

As nice as python manage.py syncdb is when the project is getting started and is still in early phases, it becomes less and less useful, mostly in handling schema changes. Since syncdb only creates a model if it doesn’t already exist in the database, we found ourselves outputting sql with python manage.py sql APP_NAME and then making adhoc ALTER TABLE scripts from the CREATE statements and passing them around to each other and then trying to remember the order to apply them in production.

Furthermore, syncdb left us without a way to alter data, add indexes, remove indexes, etc. Granted that wasn’t it’s intent and I am certainly not trying to beat up syncdb — it does what it does well.

The Solution

In order to address this, it seemed the most natural thing was to have some structured and ordered way to write SQL scripts that were applied in a uniform manner so that it was repeatable. This way we could run on our development databases, on test databases, and in deployment on production all the same way.

First step to achieve this was to version the database. This is accomplished with a simple table that will keep track of what it has executed along with some other metadata primarily for reference purposes.

CREATE TABLE `versions` (
    `version` VARCHAR(200) NOT NULL, 
    `date_created` DATETIME NOT NULL,
    `sql_executed` LONGTEXT NULL,
    `svn_version` int null
);

In order to populate this table and keep versions consistent, the naming scheme of the sql scripts should follow something that lends itself to easy sorting:

YYYYMMDD-##.sql

where pound signs are zero-padded integers of the changes for the day. In a team environment, it is probable that two people might be working on a database change at the same time and therefore would be use 01. Whoever committed first would be able to keep the 01, the other developer would get a conflict message and need to rename his script to 02.

These scripts live in a db/ folder in our project. The name of this folder is not important, but it is important to keep all the sql scripts in a folder not mixed in with other files.

To tie it all together there is s simple python script that gets the list of applied changes from the database, gets a list of files in the current directory in order, filters out the ones that have been applied and then processes the remaining scripts. In processing the scripts, it opens each script and splits script down into individual statements (splitting on the semicolon). After execution of an entire script file is complete, the versions table is updated to reflect the applied version.

Running python upgrade.py by itself will simply print to standard output all the statements it plans to execute, so that they can be reviewed. Running python upgrade.py --execute will actually execute the scripts.

If an error was found in executing a statement, processing stops immediately.

You can find upgrade.py in Django Snippet 849.

Improvements

There are lots of improvements to this script that I’d like to add if I got around to it. The ones I’d like to see most is support for “rollbacks”. Comments and suggestions are most welcome!

Introduction

I’ve recently (past week or so) been digging into the newforms-admin branch of django. I am really looking forward to this code line getting merged into trunk as there are tons of great work in this branch. I especially like the separation of the admin code from the models.

Well, I’ll get to my point.

The Problem

I needed to be able to edit child records in the admin that were children through the use of GenericRelations. I was told about a solution on Google Code called django-genericadmin, but it seemed dead and I could not get it to work.

The thing that seemed the closest to working was a patch on ticket 4667 by Honza Kral and a variation on this patch in the Django Snippet 765. Neither of these worked for r7771 of branches/newforms-admin.

The Solution

I put the 765 snippet in a file called generic.py in the root of the django project I was working on and fiddled with it until I got it working. I ended up needing to change an import, fix some variable names, and change the argument list order in one of the init methods. I also added the canorder and candelete options to the GenericInlineModelAdmin class.

After I got it working isolated in generic.py, I then added the code to the django/contrib/admin/options.py file and created a new patch (after testing locally of course). This patch has since been added to ticket 4667. I also posted the full version that I had running in generic.py in snippet 832 in case someone wants to use it without patching their newforms-admin branch.

Example of How to Use

In order to get a sense for how to make use of this functionality, imagine you have a songs app that you use to model and record details about songs, you have built this generically so you can reuse it wisely in a number of different projects. One project in particular has a need to to reference song data.

You are building a home media inventory system and decide you want to catalog both your albums as well as your DVD collection, however, you want the ability to record what songs exist on both your albums as well as your DVDs as you are a fanatic about soundtracks.

So you have the following models (abbreviated for clarity):

# Media Collection App
class Album(models.Model):
    ...

class DVD(models.Model):
    ...

# Song App
class Song(models.Model):
    ...

Since you want to enable your Song model to relate to two other models generically, you decide to use Generic Relations like so:

from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic

class Song(models.Model):
    ...
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

and then on the models that you want to have 0 to Many child records that are songs:

from myproject.song.models import Song
from django.contrib.contenttypes import generic

class Album(models.Model):
    ...
    songs = generic.GenericRelation(Song)

class DVD(models.Model):
    ...
    songs = generic.GenericRelation(Song)

Now that’s all you need to do if you didn’t care about editing the data in the admin, however, since that is not the point of the article, here is what you need to do in your admin.py in the app that contains Album and DVD:

...
from myproject.song.models import Song
from myproject.generic import GenericTabularInline
### OR ###
### from django.contrib.admin.options import GenericTabularInline

class SongInline(GenericTabularInline):
    model = Song
    extra = 2 
    ct_field_name = 'content_type'
    id_field_name = 'object_id'

class AlbumOptions(admin.ModelAdmin):
    model = Album
    inlines = (SongInline,)

class DVDOptions(admin.ModelAdmin):
    model = DVD
    inlines = (SongInline,)

admin.site.register(Album, AlbumOptions)
admin.site.register(DVD, DVDOptions)

Now you should be able to add/remove songs when editing an Album or DVD in your admin.

Update: Fixed typo in code sample above in setting the GenericForeignKey() — thanks FunkyBob!

So on one of our test servers I was doing some heavy performance testing when I noticed a strange OperationalError exception message in the Debug console for Django:

I went digging around and sure enough I had all kinds of locks when I attempted:

mysql> SHOW PROCESSLIST;

So next I checked the status of the free space available to learn than the partition that the mysql data was on was completely full. So, shutdown mysql and relocated the data to another partition with plenty of room:

mysqladmin shutdown
mv /usr/local/apache/var/* /new/location/for/files/

Then I changed /etc/my.cnf so that it had the following lines edited or amended:

[mysqld_safe]
datadir = /new/location/for/files/

Make sure that the ib_logfile files don’t remain in the new location:

rm /new/location/for/files/ib_logfile*

Now restart mysql and verify that everything is working:

mysqld_safe &
echo "create database testdatabase;" | mysql
echo "show databases;" | mysql

Not to say that this is a bug in django but rather a bug in how I had written some of my models.

Just wanted to point out a quick tip for anyone else who may have done the same thing as me as well as for any of the django experts out there that might suggest a better way.

What was happening was that I began to notice a wild discreptancy between two datetime columns in several models. They should have been identical, however, they were both getting their dates differently:

...
date_column_one = models.DateTimeField(default=datetime.now())
date_column_two = models.DateTimeField(auto_now_add=True)
...

What seems to be happening with this configuration is that datetime.now() is executed once when the server starts (and/or the module is imported) instead of what I thought would happen when I initially wrote these and that datetime.now() would get executed anytime an instance of the model was created and saved as a new record.

My solution was to override the save method:

...
date_column_one = models.DateTimeField()
...
def save(self):
    if not self.id:
        self.date_column_one = datetime.now()
    super(MyModel, self).save()
...

I am sure there is probably a better way and if you know of one, please leave a comment or send me an email and let me know!

In managing releases to a production django project, I find myself using python manage.py sqlall APPNAME to put the sql needed in a script for any new models, and then writing the ALTER or DROP statements by hand to also go into the sql scripts.

One thing that not running syncdb on production leaves out are the creation of the content type and permission records (if using django.contrib.auth and django.contrib.contenttypes). It was recommended to me on #django that I look at each of these apps management.py module to see where they hook into the post_syncdb signal.

In doing so I discovered some methods that I could just call from python and keep both my permissions and content types up to date:

from django.core.management import setup_environ
try:
    import settings
except ImportError:
    import sys
    sys.stderr.write("Couldn't find the settings.py module.")
    sys.exit(1)

setup_environ(settings)

# Add any missing content types
from django.contrib.contenttypes.management \
    import create_all_contenttypes 
create_all_contenttypes()

# Add any missing permissions
from django.contrib.auth.management import create_permissions
from django.db.models import get_apps
for app in get_apps():
    create_permissions(app, None, 2)

Enjoy.

Oh how I long to move this blog off of WordPress and onto something I build (or cull together) on top of the Django platform. It feels like it is mostly pulling together a host of different apps that are already written:

• django-tagging
• Comments
• Comment Utils
• django-pingback
• Markup
• Syndication

That leaves a core part of the blog to write — something to store the actual entries and then a URL scheme to display different levels of archives and individual posts. Of course, this schemes needs not to break current links.

This led me to check out what already might exist on Google Code and found these (limited to the most active):

• waggly-blog
• blogmaker
• django-basic-blog
• django-diario
• blogango

All are basically the same, with slightly different inclusions and approaches to this core “Blog” feature.

Lastly, a feature I’d want to implement (if there isn’t already an app out there supporting it that I can integrate) is a MetaWeblog API interface so that I can use MarsEdit, my favorite blog publish. Then all that would be left would be figuring out the data structure of WordPress and migrate over all my content (this is the only part that I am not looking forward to).

This is a really cool site for getting a sense for the django community in your community.

I just put my profile up. We need a micro-format (maybe hcard would work) for “profiles” so the same set of metadata about ourselves can just be reused at the different “profile sites” that we find ourselves joining.