Topp Engineering

I’m about to head off on vacation, so this seemed like as good a time as any to kick this out of my drafts folder… 

As some of you know, I’ve been doing the brainpower project as a Django admin application.

The reasons for this decision were:

• Django admin is touted as a very quick way to build CRUD applications, since it generates a UI from your model that in many cases is good enough for end users. No forms to write, maybe just a little tweaking. Brainpower is nothing other than a simple CRUD application, so this sounded like a perfect match.
  
• Good excuse to learn a little about Django.
  
• Get me to do something other than Zope for once in my freakin’ life.

So how did that pan out?

Well, in one month since our first requirements-gathering meeting with our “customer” (Liz and Robin from Streetfilms), in addition to all the other stuff I’ve been doing, I built something they said was good enough to start using. I did in really just a week or so of work, alone, from scratch, with almost zero advance knowledge of Django. I even spent some time testing and tuning performance (just enough that I feel confident we won’t ever have a problem with it). This also includes a full suite of Flunc tests; a random content generation script that I used for the performance tests; and a build script for development & deployment using Fassembler.

The core code is tiny: the bulk of it is in a 250-line models.py module that is reasonably clean.

As usual, writing the core code is only a small part of the story. A large portion of my time went to things like figuring out how to conveniently run external functional tests against django with a scratch database, writing and fixing the build script, and troubleshooting my initial attempts at deployment (tripping on a django bug.).

I do have some general early impressions of Django.

Things I liked

• The admin interface is, indeed, pretty slick (with a few minor oddnesses like a pretty useless Time widget).
• If I had another little CRUD app to build that seemed well-matched for Django, I could probably do it ridiculously fast.
• The Django docs are pretty decent for the most part, much better than the current state of, say, Zope 2 docs, and more extensive and thorough than the Pylons docs. (Too bad I had to quickly un-learn a bunch of stuff when I had to switch to developing against more recent django checkouts.)
• The stable release’s way to create an admin UI — by writing an inner class named Admin inside your Model — smelled really bad to me. Thankfully, this is gone in the newforms-admin branch. Similarly, you used to wire up custom validators inside your Model class and do cleanup in its save() method; now you can do custom validation in a ModelForm subclass, and you can do data cleanup in the same place. Newforms-admin is pretty nice!
  
• Got a multiple choice field that you need users to be able to extend with new choices?  Just add a foreign key field to your model referencing another model, and it just plain works in exactly the way you’d hope.  Slick!
  

Gripes

• I wish they had used an existing template language, I don’t see anything compellingly different about Django’s.
  
• I wish Django’s setup.py had a “develop” command.
• I wish the tools for syncing your models to your database were more developed. You seem to be entirely on your own for migrations. For example, if you modify the type of a field in a Model, you’ll likely need to either drop your tables and recreate them, or if you have production data to preserve, either do a dump-modify-restore cycle or hand-hack the database in place to keep your app working. At one point I ended up needing to do a dump-modify-restore using xml exports and lxml transforms; it took longer than I would have liked and I might try another strategy next time. For other model modifications such as adding a field, you might be able to get away with re-running manage.py syncdb; unfortunately I don’t know how to predict when this will just work and when it won’t.
• I wish all Django manage.py operations could be performed non-interactively. For example, for my flunc tests I wanted to reliably create and destroy a scratch database with a test admin user. Django provides a way to do most of this, but the commands that create an admin user must be run interactively. I tried using a database fixture, but that didn’t work reliably — I’m wildly guessing there’s some salt that gets reset and I don’t know when? I ended up having my test script use pexpect to drive the interactive commands.
• It seems I arrived at an inconvenient time. Django 0.96.2 may be the “latest stable release”, but it’s aging. The developers are all focused on a “newforms-admin” branch, which is cleaner and more extensible, but this work hasn’t landed on trunk yet. I don’t know when that will happen, or what else will change before another stable release. I opted to develop against the old stable release. I thought I was being smartly conservative :-p Unfortunately I soon hit a really irritating bug. While trying to understand the admin code enough to find a workaround (and feeling like I was on a familiar learning curve), I was advised to check if trunk still had the same problem, which meant an hour or two adjusting my code to work against trunk. Trunk turned out to fix only half of the problem. Then a week later the bug got fully fixed… on the newforms-admin branch. So much for trying to stick to a stable release! Maybe I was just unlucky. If I didn’t need to search fields from a join, I wouldn’t have hit this bug. (But isn’t that a common need?)
  
• Customizing and extending the admin UI is commonly done in a way that also seems vaguely familiar: You just make a copy of the thing you want to modify and hack away on it. I’ve already had a hint of pain with this — my two modified template copies broke on both Django upgrades. That’s how things typically go in the Plone world: Every single skin override you did on a Plone site would add a drop of future pain to every Plone upgrade you ever tried to do.

  This is a hard problem to solve. Just as Plone did, Django admin seems to be gradually adding more plug-in extension points so you don’t have to override the core templates as often; instead you just flip some configuration switch and/or add another template that magically gets slurped into the right place. Which has its own headaches, as every one of those bits of flexibility adds to the learning curve.  Let’s hope Greenspun’s Tenth doesn’t grow a corollary substituting Plone for Lisp :-]
  

Last week I attended the Chicago Google App Engine Hack-A-Thon, which was a small day-long event to hang out at the Chicago Google offices and work on things related to App Engine.

I was hoping for something structured more like a coding sprint, where people would work together on small projects. I think some other people were also expecting something like this, but the day kind of wandered by without any clear “let’s start doing stuff together” moment, and the opportunity was lost. Some people followed a tutorial given by Marzia Niccolai, some worked on their own projects, and there was a bit of chatting.

I started working on a simple project I’d started to make a very (very) simple CMS/Wiki. After playing with Javascript for a little while I was chatting about wikis and MoinMoin came up. After a quick look at the code I realized it was very tightly bound to the filesystem, and lacked even the most basic abstraction layer for storage. MoinMoin should have an abstraction layer, but I wasn’t really inclined to work on that. Instead I thought: why not try a fake writable filesystem?

I knew from the start that the idea was a bit absurd. The App Engine data store isn’t much like a filesystem. Files are quick to read and write, okay for scanning, hard to query. The data store is okay to read and write, very slow to scan, and easy to query. But I figured there was some hack value to it.

I don’t know that I really accomplished my goal, but I did get some code in appengine-monkey. It kind of worked, but my strategy was probably wrong: I simulated an entire filesystem, except those places that were mingled with code (where templates would typically be stored). Instead I should have required explicit locations that would be handled by the data store (e.g., /wiki-data). Python doesn’t expect file operations to lead to a lot of Python routines, and there were some circular situations deep in the Python code as a result. I seemed to mostly work those out, but didn’t actually get MoinMoin to render anything, I only got it to work very hard on the slow process of setting up its files.

I’ve been a little annoyed with the App Engine environment from the start, because it left out lots of routines that are present in every other environment (e.g., os.utime). You can’t use these routines in App Engine because there is no file writing. But it would all be more sensible if every call to these routines just raised a permission error. This is the kind of error that existing code understands. Instead you can’t even import the routines, making App Engine incompatible with lots of existing code. That would be fine except the incompatibility is so trivial. appengine-monkey seeks to relieve some of this, but it would be much simpler if it was just there in the platform to start with.

­Conclusion? Porting code is hard, but porting old, organically developed code like MoinMoin is really hard. And App Engine still needs a good wiki.

­­­­Robert Marianski and I have been working on adding the ability to import and export data from listen mailing lists^*. Specifically, we’ve been working on exporting mailing list subscribers to CSV files, exporting messages as mbox files, and importing mbox files into the mailing list archive.

By far the most interesting of these is importing mbox files into the mailing list archive. First (along with exporting) it allows for a new use case: Transitioning from one list to another. This gives users more flexibility and helps to avoid lock-in.

Second, it presents some challenging design problems. Importing can be seen in some respects as a relatively “dangerous” operation, as in the process it’s possible to accidentally import the wrong file or even import a corrupt file (this is in fact is something that’s gotten me in the past while migrating from one mail client to another; my “solution” involved manually deleting a lot of messages and then editing the source of the mbox file to fix the errors, hardly something that most users would ever want to do). In either case, your mailing list archive is now polluted with a bunch of messages you don’t want there. Worse, listen doesn’t currently support deleting messages from the archive, meaning once messages are in the archive, there’s no getting them out.

The question is, how do you prevent this from happening? One way to at least mitigate the issue is to give a confirmation warning before actually importing the messages into the archive. This may help in the event that the user accidentally selected the wrong file, but it’s hardly ideal and it’s still quite possible something could go wrong. Even better would be to show a preview of the messages that would be imported and let the user look at them to make sure they are correct. But this brings with it its own set of questions: How large should the preview be? Should the full messages be shown or just some of the headers? Do you make the user submit the file twice (once initially, once again when confirming) or do you store it on the server? What happens if the user never clicks anything on the confirmation page? What happens when the user confirms and then changes his mind?

Rob and I quickly realized that this was a case of using a warning when we really meant undo. What would be most useful for a user would be to be able to undo any import — at any time.

From a technical standpoint, this is actually not too complicated (save for a few nasty and hard to isolate Zope bugs, which I won’t bore you all with here…). All we had to do was store the message IDs, along with some other meta data (e.g., the time of the import), and provide an interface for removing just those messages from the archive. And this is exactly what we did: You can now remove any messages associated with any past import, even if you’ve subsequently imported other messages after them.

 The branch we worked on is here, and here’s a screenshot to give you a better idea of what we did:

* There were a few days a couple weeks ago that I’ve missed chronicling so far. During those days Anil and I worked on the Feed Me plugin for WordPress, which makes it easy to incorporate feeds from Delicious, Flickr, and YouTube into any WordPress blog. The plugin has been used by Streetsblog for a while , but we generalized it and added a handy admin interface that lets you easily customize the number and type of feeds used. Check out the project page.