Entries Tagged 'development' ↓

In my continued experimentation with the newforms-admin branch of Django, I wanted to figure out how to order fields of an inline. Looking at the documentation for inlines I saw there was not an ordering field.

I had thought that there was but it turns out I was just mistaken in the object hierarchy. InlineModelAdmin inherits from BaseModelAdmin the same as ModelAdmin — I was thinking that InlineModelAdmin inherited from ModelAdmin.

Therefore, I had to determine a way to accomplish this via some spelunking through the code. At first, I thought I’d just create my own template by inheriting or copying the tabular.html template. After looking at it, I determined that I didn’t want to figure out how to do a regroup on inlineadminformset or if even that was the proper way to try to order things in that template.

After a few minutes in the code I realized that I could just subclass BaseInlineFormset and specify the fields that I wanted to order by (in this example I will use starttime and endtime:

from django.contrib import admin
from django.newforms.models import BaseInlineFormset

class MyOrderedFormset(BaseInlineFormset):
    def get_queryset(self):
        qs = super(SessionInlineFormset, self).get_queryset()
        return qs.order_by('start_time', 'end_time')

class MyOrderedInline(admin.TabularInline):
    model = MyModel
    extra = 1
    formset = MyOrderedFormset

That’s it. Now after that section of inlines are ordered by starttime and endtime.

Introduction

Managing database changes in a team environment working on a django project can be complicated. I would imagine that there is no one size fits all solution and it would depend on team size and configuration, production database size, etc.

What I will outline here is a solution I developed for a small team that I work with on a django based web application. So far it has worked really well for allowing us to streamline database changes so we can stay in sync, deploy easily, and add small tweaks to the database (indexing, data manipulation, etc.) that is semi-automated.

The idea is based partly on Rails database migrations (though not has sleek and well put together) and a database versioning system that was used on a team that I have worked on previously using MS SQL in a corporate environment and a fairly decent team size. In one sense it is not nearly as well put together as either of these two solutions, but at the same time, it is pretty much hands off and has been working well for us for a number of months now.

Why not just use syncdb?

As nice as python manage.py syncdb is when the project is getting started and is still in early phases, it becomes less and less useful, mostly in handling schema changes. Since syncdb only creates a model if it doesn’t already exist in the database, we found ourselves outputting sql with python manage.py sql APP_NAME and then making adhoc ALTER TABLE scripts from the CREATE statements and passing them around to each other and then trying to remember the order to apply them in production.

Furthermore, syncdb left us without a way to alter data, add indexes, remove indexes, etc. Granted that wasn’t it’s intent and I am certainly not trying to beat up syncdb — it does what it does well.

The Solution

In order to address this, it seemed the most natural thing was to have some structured and ordered way to write SQL scripts that were applied in a uniform manner so that it was repeatable. This way we could run on our development databases, on test databases, and in deployment on production all the same way.

First step to achieve this was to version the database. This is accomplished with a simple table that will keep track of what it has executed along with some other metadata primarily for reference purposes.

CREATE TABLE `versions` (
    `version` VARCHAR(200) NOT NULL, 
    `date_created` DATETIME NOT NULL,
    `sql_executed` LONGTEXT NULL,
    `svn_version` int null
);

In order to populate this table and keep versions consistent, the naming scheme of the sql scripts should follow something that lends itself to easy sorting:

YYYYMMDD-##.sql

where pound signs are zero-padded integers of the changes for the day. In a team environment, it is probable that two people might be working on a database change at the same time and therefore would be use 01. Whoever committed first would be able to keep the 01, the other developer would get a conflict message and need to rename his script to 02.

These scripts live in a db/ folder in our project. The name of this folder is not important, but it is important to keep all the sql scripts in a folder not mixed in with other files.

To tie it all together there is s simple python script that gets the list of applied changes from the database, gets a list of files in the current directory in order, filters out the ones that have been applied and then processes the remaining scripts. In processing the scripts, it opens each script and splits script down into individual statements (splitting on the semicolon). After execution of an entire script file is complete, the versions table is updated to reflect the applied version.

Running python upgrade.py by itself will simply print to standard output all the statements it plans to execute, so that they can be reviewed. Running python upgrade.py --execute will actually execute the scripts.

If an error was found in executing a statement, processing stops immediately.

You can find upgrade.py in Django Snippet 849.

Improvements

There are lots of improvements to this script that I’d like to add if I got around to it. The ones I’d like to see most is support for “rollbacks”. Comments and suggestions are most welcome!

Introduction

I’ve recently (past week or so) been digging into the newforms-admin branch of django. I am really looking forward to this code line getting merged into trunk as there are tons of great work in this branch. I especially like the separation of the admin code from the models.

Well, I’ll get to my point.

The Problem

I needed to be able to edit child records in the admin that were children through the use of GenericRelations. I was told about a solution on Google Code called django-genericadmin, but it seemed dead and I could not get it to work.

The thing that seemed the closest to working was a patch on ticket 4667 by Honza Kral and a variation on this patch in the Django Snippet 765. Neither of these worked for r7771 of branches/newforms-admin.

The Solution

I put the 765 snippet in a file called generic.py in the root of the django project I was working on and fiddled with it until I got it working. I ended up needing to change an import, fix some variable names, and change the argument list order in one of the init methods. I also added the canorder and candelete options to the GenericInlineModelAdmin class.

After I got it working isolated in generic.py, I then added the code to the django/contrib/admin/options.py file and created a new patch (after testing locally of course). This patch has since been added to ticket 4667. I also posted the full version that I had running in generic.py in snippet 832 in case someone wants to use it without patching their newforms-admin branch.

Example of How to Use

In order to get a sense for how to make use of this functionality, imagine you have a songs app that you use to model and record details about songs, you have built this generically so you can reuse it wisely in a number of different projects. One project in particular has a need to to reference song data.

You are building a home media inventory system and decide you want to catalog both your albums as well as your DVD collection, however, you want the ability to record what songs exist on both your albums as well as your DVDs as you are a fanatic about soundtracks.

So you have the following models (abbreviated for clarity):

# Media Collection App
class Album(models.Model):
    ...

class DVD(models.Model):
    ...

# Song App
class Song(models.Model):
    ...

Since you want to enable your Song model to relate to two other models generically, you decide to use Generic Relations like so:

from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic

class Song(models.Model):
    ...
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

and then on the models that you want to have 0 to Many child records that are songs:

from myproject.song.models import Song
from django.contrib.contenttypes import generic

class Album(models.Model):
    ...
    songs = generic.GenericRelation(Song)

class DVD(models.Model):
    ...
    songs = generic.GenericRelation(Song)

Now that’s all you need to do if you didn’t care about editing the data in the admin, however, since that is not the point of the article, here is what you need to do in your admin.py in the app that contains Album and DVD:

...
from myproject.song.models import Song
from myproject.generic import GenericTabularInline
### OR ###
### from django.contrib.admin.options import GenericTabularInline

class SongInline(GenericTabularInline):
    model = Song
    extra = 2 
    ct_field_name = 'content_type'
    id_field_name = 'object_id'

class AlbumOptions(admin.ModelAdmin):
    model = Album
    inlines = (SongInline,)

class DVDOptions(admin.ModelAdmin):
    model = DVD
    inlines = (SongInline,)

admin.site.register(Album, AlbumOptions)
admin.site.register(DVD, DVDOptions)

Now you should be able to add/remove songs when editing an Album or DVD in your admin.

Update: Fixed typo in code sample above in setting the GenericForeignKey() — thanks FunkyBob!

Until I no longer have to work with Subversion (or subsequent releases of Subversion improve), dealing with merges will remain a hassle. Oh, branching is super easy (just a simple server side copy command) but for those branches to be worth anything you need to be able to merge efficiently.

Yes, I worked with the svnmerge.py hack for a couple of months, but it was introducing strange merge bugs and just generally left me uncomfortable trusting to do all its magic (relying on certain property values to guide its merge operations).

In light of going back to the old style svn merge, where I relied on comments to tell me the last merge point, I wanted to make sure that the log messages were standardized. Without some tool to help me do this, I was spending most of the merge time formatting comments and they ended up not being a reliable standard.

Taking advantage of the ability to output Subversion log messages into xml, I wrote a quick python script to generate single line summaries of the revision history:

# slfmt.py LOGFILE
from xml.dom.minidom import parse
import sys

doc = parse(sys.argv[1])

revisions = doc.getElementsByTagName('logentry')

for rev in revisions:
    r = rev.getAttribute('revision')
    try:
        msg = rev.getElementsByTagName('msg')[0].childNodes[0].data.split('\n')[0]
        print 'r%s - %s' % (r, msg)
    except IndexError:
        print 'r%s - NO LOG MESSAGE' % r

Then I wrapped up several other commands I find myself constantly running into a single bash script:

#!/bin/bash
# prepmerge REV BRANCH
REPO_ROOT=REPLACE_WITH_YOURS

svn merge -r$1:HEAD $REPO_ROOT/$2
echo "Merged r$1:HEAD from $2 into "`svn info | grep URL | awk '{ print $2 }'` > .merge-comment
echo "" >> .merge-comment
echo "  svn info:" >> .merge-comment
svn info $REPO_ROOT/$2 | awk '{ print "\t\t"$0 }' >> .merge-comment
echo "  change log:" >> .merge-comment
svn log --xml -r$1:HEAD $REPO_ROOT/$2 > .log.xml
python ~/bin/slfmt.py .log.xml | awk '{ print "\t\t"$0 }' >> .merge-comment
echo "" >> .merge-comment
echo "  files:" >> .merge-comment
svn st | grep -v "?" | awk '{ print "\t\t"$0 }' >> .merge-comment

With these two scripts in my ~/bin directory and assuming that my repo follows the standard /trunk, /branches, /tags, model, I can now do merges in three simple steps. Assuming you would like to merge changes from a branch called experimental from HEAD into trunk, you would need a local working copy of trunk and be in the the root of that local path and then:

• ~/bin/prepmerge HEAD branches/experimental
• review changes, resolve conflicts
• svn commit -F .merge-comment

That’s it. I am sure there are better/cleaner methods to this, but this now allows me to focus solely on resolving conflicts that emerge instead of fiddling with comments. Of course, if everything I worked on was on git, I wouldn’t have to fool with these types of scripts.

Well, Ben stole my thunder and posted about it first. I’ll go ahead an take the liberty and reiterate the news here as well.

We finally shipped Lysine 1.0, Amino Software’s EBCDIC Source Component for SSIS. With this component you can read the raw binary data dumps from mainframes directly into Microsoft’s SQL Server via SSIS. This should eliminate the need for most cases of having to write preprocessors to transform the data prior to integration.

In summary, Lysine offers:

• Imports Raw Binary EBCDIC Data (no preprocessing in most cases)
• Several EBCDIC code pages supported
• Intuitive Layout UI for rapid development
• Quick Preview to show you if your layout is correct
• All major column types supported:
  • Redefines
  • Occurs
  • Occurs Depending On
  • Packed (Comp-3), Zoned
• Single Pass conversion for scalable performance
• Export/Import of layout for team development and versioning

There is a fully functional 30-day trial available along with a recently writing user guide and an array of support options now available on http://aminosoftware.com.

Not to say that this is a bug in django but rather a bug in how I had written some of my models.

Just wanted to point out a quick tip for anyone else who may have done the same thing as me as well as for any of the django experts out there that might suggest a better way.

What was happening was that I began to notice a wild discreptancy between two datetime columns in several models. They should have been identical, however, they were both getting their dates differently:

...
date_column_one = models.DateTimeField(default=datetime.now())
date_column_two = models.DateTimeField(auto_now_add=True)
...

What seems to be happening with this configuration is that datetime.now() is executed once when the server starts (and/or the module is imported) instead of what I thought would happen when I initially wrote these and that datetime.now() would get executed anytime an instance of the model was created and saved as a new record.

My solution was to override the save method:

...
date_column_one = models.DateTimeField()
...
def save(self):
    if not self.id:
        self.date_column_one = datetime.now()
    super(MyModel, self).save()
...

I am sure there is probably a better way and if you know of one, please leave a comment or send me an email and let me know!

I’ve been digging into Cocoa to learn how to write native applications on the Mac. Objective-C is actually pretty nice, once I get past the funky bracket syntax and the notion of sending messages instead of calling functions.

I have struggled to find good sources of help and information on the web. Maybe that is in part because this is a pretty narrow segment of developers (versus something like Ruby, Python, or C#). That being said, I have found pretty fast and good help can be found on the cocoa-dev mailing list.

I first set out to learn about and build a preference pane to capture configuration values that I want for the application I am building (more on that to come once I am ready to unveil what I have planned). In doing so I found some resources that helped quite a bit, yet left a little to be desired.

First the documentation was written for 10.4, but I was working with 10.5, so some of the screenshots were throwing me off. Second, there was one detail in particular that was troubling me and if it weren’t for help from a John Stiles from the cocoa-dev listserv, I would still be spinning.

The one little trick that was keeping me spinning was un-checking “Release When Closed” on the Panel Attributes inspector page:

If you don’t do that then all will work well until you got to select the application menu after closing the preference pane window. The application will freeze and then crash. Makes perfect sense now that it was pointed out to me.

There is a new book on Peepcode that covers git pretty deeply and in a way that is more easily digestible for the non-Linux hacker. It also comes with quite a bit of video demonstrating concepts in the book. Both very high quality items are sold together for a very low $9.

This graphic from page 10 of the book made me smile:

After some tireless searching for a git command that would yield a listing of local commits that have not been pushed to a target, I popped into the #git IRC channel and posed the question. I was promptly answered by Mikachu:

git log origin..

Yes, that command includes the two periods.

Wikipedia defines Duck Typing as:

In computer programming, duck typing is a style of dynamic typing in which an object’s current set of methods and properties determines the valid semantics, rather than its inheritance from a particular class. The name of the concept refers to the duck test, attributed to James Whitcomb Riley, which may be phrased as follows:

“If it walks like a duck and quacks like a duck, I would call it a duck.”

Coming from a strong background in C#, it took awhile for this feature of Python to seem useful. I was use to creating interfaces and creating a proper inheritance chain implementing these interfaces and/or defining abstract base classes, all to have a good set of mock objects to use in unit tests.

I really love the ability to quickly whip out a duck typed class in Python to make simple mock objects so that I am testing my code in a unit test and not boundary objects (e.g. urllib). I don’t need to test whether my request is properly handled by the server at the urllib endpoint and that the response is properly read and processed. I need to instead make sure that my request conforms to a published specification and that the expected result/response from the server is processed properly. This also allows my tests to run offline.

Here is an example of some mock objects that I use in pyphanfare:

# pyphanfare/tests/mocks.py
from pyphanfare.tests.testdata import data

class URLopener:
    def open(self, url):
        params = url.split('?')[1].split('&') 
        for param in params:
            tokens = param.split('=')
            if tokens[0].strip() == 'method':
                return URLhandle(tokens[1].strip())
    def addheader(self, *args):
        self.headers = args

class URLhandle:
    def __init__(self, method):
        self.method = method
    def read(self):
        return data[self.method]

pyphanfare.tests.testdata is just a Python module that contains test data in the form of a dictionary object indexed by method name that I am calling on the API having a value of XML that is expected in the response.