Topp Engineering

I’m about to head off on vacation, so this seemed like as good a time as any to kick this out of my drafts folder… 

As some of you know, I’ve been doing the brainpower project as a Django admin application.

The reasons for this decision were:

• Django admin is touted as a very quick way to build CRUD applications, since it generates a UI from your model that in many cases is good enough for end users. No forms to write, maybe just a little tweaking. Brainpower is nothing other than a simple CRUD application, so this sounded like a perfect match.
  
• Good excuse to learn a little about Django.
  
• Get me to do something other than Zope for once in my freakin’ life.

So how did that pan out?

Well, in one month since our first requirements-gathering meeting with our “customer” (Liz and Robin from Streetfilms), in addition to all the other stuff I’ve been doing, I built something they said was good enough to start using. I did in really just a week or so of work, alone, from scratch, with almost zero advance knowledge of Django. I even spent some time testing and tuning performance (just enough that I feel confident we won’t ever have a problem with it). This also includes a full suite of Flunc tests; a random content generation script that I used for the performance tests; and a build script for development & deployment using Fassembler.

The core code is tiny: the bulk of it is in a 250-line models.py module that is reasonably clean.

As usual, writing the core code is only a small part of the story. A large portion of my time went to things like figuring out how to conveniently run external functional tests against django with a scratch database, writing and fixing the build script, and troubleshooting my initial attempts at deployment (tripping on a django bug.).

I do have some general early impressions of Django.

Things I liked

• The admin interface is, indeed, pretty slick (with a few minor oddnesses like a pretty useless Time widget).
• If I had another little CRUD app to build that seemed well-matched for Django, I could probably do it ridiculously fast.
• The Django docs are pretty decent for the most part, much better than the current state of, say, Zope 2 docs, and more extensive and thorough than the Pylons docs. (Too bad I had to quickly un-learn a bunch of stuff when I had to switch to developing against more recent django checkouts.)
• The stable release’s way to create an admin UI — by writing an inner class named Admin inside your Model — smelled really bad to me. Thankfully, this is gone in the newforms-admin branch. Similarly, you used to wire up custom validators inside your Model class and do cleanup in its save() method; now you can do custom validation in a ModelForm subclass, and you can do data cleanup in the same place. Newforms-admin is pretty nice!
  
• Got a multiple choice field that you need users to be able to extend with new choices?  Just add a foreign key field to your model referencing another model, and it just plain works in exactly the way you’d hope.  Slick!
  

Gripes

• I wish they had used an existing template language, I don’t see anything compellingly different about Django’s.
  
• I wish Django’s setup.py had a “develop” command.
• I wish the tools for syncing your models to your database were more developed. You seem to be entirely on your own for migrations. For example, if you modify the type of a field in a Model, you’ll likely need to either drop your tables and recreate them, or if you have production data to preserve, either do a dump-modify-restore cycle or hand-hack the database in place to keep your app working. At one point I ended up needing to do a dump-modify-restore using xml exports and lxml transforms; it took longer than I would have liked and I might try another strategy next time. For other model modifications such as adding a field, you might be able to get away with re-running manage.py syncdb; unfortunately I don’t know how to predict when this will just work and when it won’t.
• I wish all Django manage.py operations could be performed non-interactively. For example, for my flunc tests I wanted to reliably create and destroy a scratch database with a test admin user. Django provides a way to do most of this, but the commands that create an admin user must be run interactively. I tried using a database fixture, but that didn’t work reliably — I’m wildly guessing there’s some salt that gets reset and I don’t know when? I ended up having my test script use pexpect to drive the interactive commands.
• It seems I arrived at an inconvenient time. Django 0.96.2 may be the “latest stable release”, but it’s aging. The developers are all focused on a “newforms-admin” branch, which is cleaner and more extensible, but this work hasn’t landed on trunk yet. I don’t know when that will happen, or what else will change before another stable release. I opted to develop against the old stable release. I thought I was being smartly conservative :-p Unfortunately I soon hit a really irritating bug. While trying to understand the admin code enough to find a workaround (and feeling like I was on a familiar learning curve), I was advised to check if trunk still had the same problem, which meant an hour or two adjusting my code to work against trunk. Trunk turned out to fix only half of the problem. Then a week later the bug got fully fixed… on the newforms-admin branch. So much for trying to stick to a stable release! Maybe I was just unlucky. If I didn’t need to search fields from a join, I wouldn’t have hit this bug. (But isn’t that a common need?)
  
• Customizing and extending the admin UI is commonly done in a way that also seems vaguely familiar: You just make a copy of the thing you want to modify and hack away on it. I’ve already had a hint of pain with this — my two modified template copies broke on both Django upgrades. That’s how things typically go in the Plone world: Every single skin override you did on a Plone site would add a drop of future pain to every Plone upgrade you ever tried to do.

  This is a hard problem to solve. Just as Plone did, Django admin seems to be gradually adding more plug-in extension points so you don’t have to override the core templates as often; instead you just flip some configuration switch and/or add another template that magically gets slurped into the right place. Which has its own headaches, as every one of those bits of flexibility adds to the learning curve.  Let’s hope Greenspun’s Tenth doesn’t grow a corollary substituting Plone for Lisp :-]
  

Last week I attended the Chicago Google App Engine Hack-A-Thon, which was a small day-long event to hang out at the Chicago Google offices and work on things related to App Engine.

I was hoping for something structured more like a coding sprint, where people would work together on small projects. I think some other people were also expecting something like this, but the day kind of wandered by without any clear “let’s start doing stuff together” moment, and the opportunity was lost. Some people followed a tutorial given by Marzia Niccolai, some worked on their own projects, and there was a bit of chatting.

I started working on a simple project I’d started to make a very (very) simple CMS/Wiki. After playing with Javascript for a little while I was chatting about wikis and MoinMoin came up. After a quick look at the code I realized it was very tightly bound to the filesystem, and lacked even the most basic abstraction layer for storage. MoinMoin should have an abstraction layer, but I wasn’t really inclined to work on that. Instead I thought: why not try a fake writable filesystem?

I knew from the start that the idea was a bit absurd. The App Engine data store isn’t much like a filesystem. Files are quick to read and write, okay for scanning, hard to query. The data store is okay to read and write, very slow to scan, and easy to query. But I figured there was some hack value to it.

I don’t know that I really accomplished my goal, but I did get some code in appengine-monkey. It kind of worked, but my strategy was probably wrong: I simulated an entire filesystem, except those places that were mingled with code (where templates would typically be stored). Instead I should have required explicit locations that would be handled by the data store (e.g., /wiki-data). Python doesn’t expect file operations to lead to a lot of Python routines, and there were some circular situations deep in the Python code as a result. I seemed to mostly work those out, but didn’t actually get MoinMoin to render anything, I only got it to work very hard on the slow process of setting up its files.

I’ve been a little annoyed with the App Engine environment from the start, because it left out lots of routines that are present in every other environment (e.g., os.utime). You can’t use these routines in App Engine because there is no file writing. But it would all be more sensible if every call to these routines just raised a permission error. This is the kind of error that existing code understands. Instead you can’t even import the routines, making App Engine incompatible with lots of existing code. That would be fine except the incompatibility is so trivial. appengine-monkey seeks to relieve some of this, but it would be much simpler if it was just there in the platform to start with.

­Conclusion? Porting code is hard, but porting old, organically developed code like MoinMoin is really hard. And App Engine still needs a good wiki.

­­­­Robert Marianski and I have been working on adding the ability to import and export data from listen mailing lists^*. Specifically, we’ve been working on exporting mailing list subscribers to CSV files, exporting messages as mbox files, and importing mbox files into the mailing list archive.

By far the most interesting of these is importing mbox files into the mailing list archive. First (along with exporting) it allows for a new use case: Transitioning from one list to another. This gives users more flexibility and helps to avoid lock-in.

Second, it presents some challenging design problems. Importing can be seen in some respects as a relatively “dangerous” operation, as in the process it’s possible to accidentally import the wrong file or even import a corrupt file (this is in fact is something that’s gotten me in the past while migrating from one mail client to another; my “solution” involved manually deleting a lot of messages and then editing the source of the mbox file to fix the errors, hardly something that most users would ever want to do). In either case, your mailing list archive is now polluted with a bunch of messages you don’t want there. Worse, listen doesn’t currently support deleting messages from the archive, meaning once messages are in the archive, there’s no getting them out.

The question is, how do you prevent this from happening? One way to at least mitigate the issue is to give a confirmation warning before actually importing the messages into the archive. This may help in the event that the user accidentally selected the wrong file, but it’s hardly ideal and it’s still quite possible something could go wrong. Even better would be to show a preview of the messages that would be imported and let the user look at them to make sure they are correct. But this brings with it its own set of questions: How large should the preview be? Should the full messages be shown or just some of the headers? Do you make the user submit the file twice (once initially, once again when confirming) or do you store it on the server? What happens if the user never clicks anything on the confirmation page? What happens when the user confirms and then changes his mind?

Rob and I quickly realized that this was a case of using a warning when we really meant undo. What would be most useful for a user would be to be able to undo any import — at any time.

From a technical standpoint, this is actually not too complicated (save for a few nasty and hard to isolate Zope bugs, which I won’t bore you all with here…). All we had to do was store the message IDs, along with some other meta data (e.g., the time of the import), and provide an interface for removing just those messages from the archive. And this is exactly what we did: You can now remove any messages associated with any past import, even if you’ve subsequently imported other messages after them.

 The branch we worked on is here, and here’s a screenshot to give you a better idea of what we did:

* There were a few days a couple weeks ago that I’ve missed chronicling so far. During those days Anil and I worked on the Feed Me plugin for WordPress, which makes it easy to incorporate feeds from Delicious, Flickr, and YouTube into any WordPress blog. The plugin has been used by Streetsblog for a while , but we generalized it and added a handy admin interface that lets you easily customize the number and type of feeds used. Check out the project page.

This week (and by this week, I mean two weeks ago), was my first foray into the wonderful world of GIS. My initial task was to prototype a read-only data store that gets its data via a RESTful interface. Specifically, I set out to write a GeoTools data store called RESTfulDatastore that gets its data from FeatureServer.org.

I spent a lot of my initial time just trying to get up to speed and figure out what’s what in the world of GIS. Needless to say, there’s a lot to learn, and I was fabulously lost at the onset. However, thanks to lots of online documentation (and, of course, Wikipedia), I was able to get a grasp on the basics. I was also fortunate to have the help of Jody Garnett’s DataStore tutorial. Jody’s PropertyDataStore example formed the basis of RESTfulDatastore and provided a helpful framework in which I could organize my understanding of how of a data store works.

I must say, however, that this entire process was not without some frustration. On the plus side, there’s a lot of documentation both for GeoServer and GeoTools, and much of it was ultimately indispensable to me. Unfortunately, a lot of the documentation is out of date. I found myself frequently starting to code things based on examples in the docs only to find that the method I was using was deprecated; sometimes I found updated versions of the docs that explained how things should be done now, but other times I didn’t.

After getting a very rough prototype of the data store working (see ticket here, and beware, when I say rough, I mean rough), I switched my focus to writing a separate class to parse GeoJSON objects, inventively named GeoJSONParser.

(A quick aside about GeoJSON: Having read some cryptic, convoluted and incomplete specs in the past, the GeoJSON spec is breath of fresh air. It’s clear, concise, and very well written. Even though it’s a simple spec to document, it’s still impressive. You should check it out.)

GeoJSONParser is essentially a complementary class to GeoJSONBuilder: Where GeoJSONBuilder takes Geometry objects and writes out strings of GeoJSON objects, GeoJSONParser does the opposite, taking JSON strings (or json-lib’s JSONObjects) and generating Java objects. Writing GeoJSONParser went much more smoothly than hacking around with the data store, mostly because I had a much better feel for what was going on having been doing geo work for several days already. A preliminary version of GeoJSONParser (along with GeoJSONBuilder) is attached to this ticket.

 Lessons learned: GIS has a lot of acronyms, Java isn’t so bad, good documentation is indispensable.

I spent most of my second week doing some Xinha UI work with Doug. Our broad directive was to improve the Xinha user experience, so we started by talking with Sonali and Bryan about known interface bugs and quirks, and got a good idea of where some of the major usability shortcomings are. My general thoughts on WYSIWYG editors for HTML-based content are that they’re almost always mediocre and are hard to do well as full-fledged desktop applications, and are even tougher to do well as web-based apps. To date, I’ve never seen any browser-based WYSIWYG editor that didn’t require me to either fight its idiosyncricities or delve into the source to get it to do exactly what I wanted it to. Xinha has come a long way, but there’s definitely still a lot of work to be done.

We decided to start by concentrating on Xinha’s popup dialogs, which are used in plugins all over the place, e.g., for adding a link or inserting an image. Since many people find popups annoying, not to mention the fact that they interfere with people’s popup-blockers, we decided to look into converting the existing popups into lightboxes. Not wanting to reinvent the wheel, we looked at several different options for lightbox libraries. I hadn’t realized just how many lightbox libraries have (no pun intended) popped up over the past year or two. Unfortunately, many of them have significant downsides: some aren’t released under free licenses, and more than a few appear to no longer be under active development. We ultimately opted to give nyroModal a shot, since it has had more than one release, uses JQuery, is relatively full-featured, and is released under an MIT license.

Not long after we started using nyroModal, however, we came across a branch of Xinha called new-dialogs that was started a few months ago by one of the core Xinha developers. As its name suggests, the branch is a major update to Xinha’s dialog system and has had a significant amount of work put into it. Details of its development can be found in the change history for ticket #1176. In short, it does almost exactly what we were hoping to do, and overall, does it quite well.

Interestingly, new-dialogs does not use any of the aforementioned libraries for its lightboxes. Instead, it includes its own lightbox code written from scratch. Its dialogs are resizable, draggable, and even dockable, though I’m not sure how necessary (or even desirable) the first two are for a modal system.

Much of what’s left to do with new-dialogs is to transition all of the plugins that use the old popup dialogs over to the new system. Doug and I have started working on this process by updating the UnFormat plugin.  Doug has since done the BackgroundImage plugin and I’ve done the Abbreviation and FindReplace plugins. There are still quite a few plugins to go, but once the rest are updated there shouldn’t be much holding us back from making the jump to a popup-free Xinha.

The Promise

In preparation for a ramp-up in testing the OpenPlans Plone 3 upgrade efforts, I’ve been revisiting OSAF’s Windmill functional testing tool. The story is compelling; if you’re going to be launching a browser and clicking around to test a site anyway, why not turn on a recorder that will auto-generate test suites that can reproduce your actions? Even if it ends up making the original testing take a little while longer, that pays for itself the first time you re-run the tests.

With something concrete to accomplish, then, I sat down to see if Windmill delivers on this promise.

Bootstrapping

Getting started went pretty smoothly. Installing was easy, on Ubuntu Gutsy, anway; just regular setuptools stuff. I’m working from the Windmill trunk, so I created and activated a python 2.5 virtualenv, checked the code out from http://svn.osafoundation.org/windmill/trunk/, and ran ‘python setup.py develop’ to install it into the environment.

Windmill is then launchable with ‘windmill firefox URL’. This will open firefox, albeit without any of the customizations that you might have set up in your profile. It also opens a smaller Windmill IDE browser window, and starts a Windmill “controller service”, which exposes the API for manipulating the browser window.

Capturing Tests

The IDE is pretty simple. It’s implemented in HTML and Javascript, and it provides 4 primary features: a test recorder, a test runner, a DOM explorer, and an “Assert” explorer. The test recorder is what I’m first interested in. I click the record button, and the browser window jumps to the foreground. So I start clicking around, entering text and submitting forms, lo and behold, I can see what I’m doing being captured in the IDE. Good start.

To start, I make an incorrect login attempt, and, upon failure, try to create a new account, getting all the way to the “check your email” confirmation screen. I turn off the recorder, and click on the ’save’ link in the IDE window. Windmill can export the test suites as either python or JSON. I’ve chosen python, so I get another window with the following text:

# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient

def test():
    client = WindmillTestClient(__name__)

    client.click(link=u'Sign in')
    client.type(text=u'bogus', id=u'__ac_name')
    client.type(text=u'bobobobo', id=u'__ac_password')
    client.click(name=u'login')
    client.click(link=u'Create account')
    client.type(text=u'testuser1', id=u'id')
    client.type(text=u'Test UserOne', id=u'fullname')
    client.type(text=u'test1@example.com', id=u'email')
    client.type(text=u'testy', id=u'password')
    client.type(text=u'testy', id=u'confirm_password')
    client.click(name=u'task|join')

I paste this into an emacs buffer and save it as windmill_tests.py. Then I shut down the windmill process (I have to ‘kill’ it to make sure everything terminates correctly :P) and pass in the test I just created with ‘windmill firefox URL test=windmill_tests.py’. Sure enough, the browser launches, hits the site, and starts following links and entering text! Initial success makes much happy.

Controller API

The ‘client’ variable in the python code above is a handle to a WindmillTestClient object, which exposes the Windmill Controller API. The controller is what is driving the browser. One fun trick is that you can put a pdb in your test code and you’ll get an interactive prompt which you can use to control the live browser window. Now enjoy the power of a python command line interface to teh interwebs, while still being able to access all the rich multimedia that today’s users demand! It’s definitely more fun than it should be to fill out web forms and click on links by typing in ‘client.type’ and ‘client.click’ commands.

Assertions

Astute readers will have noticed that the second run of the user creation test would have a different result than the first. Indeed, when I used Windmill to run the test steps I’d captured, it ended with an error message informing me that the username was already taken. Real tests of course will want to make assertions, to ensure that the behaviour is really what you want. Enter the Assertion Explorer. At any point, you can click on the Assertion Explorer button, and then click somewhere on the page, and the IDE will generate a best-guess assertion for you for the element that you selected.

The generated assertions are sometimes spot-on. Whenever I clicked on a portal status message, for instance, it asserts (using XPath) that the PSM element exists on the page, and that it contains the specified text. When I click on regular page text, however, it only checks for the existence of the clicked node, when I want it to check the text as well.

The asserts are a part of the Controller API, and are pretty easy to understand. Here’s an example of a couple that I generated:

    client.asserts.assertNode(xpath=u'/html/body/div/div/div/div/div')
    client.asserts.assertText(xpath=u'/html/body/div/div/div/div/div', validator=u'Welcome! You have signed in.')

    client.asserts.assertNode(xpath=u’/html/body/div/div/div/div/div’)
    client.asserts.assertText(xpath=u’/html/body/div/div/div/div/div’, validator=u’Your changes have been saved.’)
    client.asserts.assertNode(link=u’something else+’) 

Nothing particularly mysterious there.

The Warts

Despite the early successes, it wasn’t long before I hit a couple bumps. Here’s an overview of the issues that came up for me:

• Hard to extract info from the page
  
  The Controller API works pretty well for controlling the page and making assertions, but there don’t seem to be very good hooks for extracting information from the page itself into the python environment. For instance, in order to complete the user registration, the test will need to log in as an admin and hit a special page which will return the user’s confirmation key. This will need to be extracted from the response and then used as part of the URL of a subsequent request. Even a simpler case, such as simply extracting the URL of the page that the browser is currently visiting, doesn’t come clear to me from a thorough comb of the documentation.
• The ‘waits’ stuff seems to be broken ATM
  
  The Controller API has a whole selection of ‘waits’ methods, which will tell the test running to not move on until some criteria is met, either a specific amount of time has elapsed, or the page has loaded, or an element shows up on the page, etc. For me, running FF2.0.0.14 on Ubuntu Gutsy, these were completely broken. Any time I try to use one of these calls, the test suite just stops right there. This is a show-stopper, since I very quickly hit false failures due to the tests running more quickly than the browser was responding.
• Xinha typing didn’t get picked up by the recorder
  
  Most of the actions I performed in the browser were dutifully recorded by the IDE. The only exception to this is any text I would enter into the Xinha editor. Whenever I would edit a wiki page, the recorder would skip directly from clicking the ‘edit’ link to clicking on the ’save’ button. I think we can work around this by just adding these commands by hand to the generated python. I’m not 100% on this, though.
  

Python is Nice

As I’ve mentioned, it’s possible to export the generated tests and assertions as either JSON or python code. Having such a straightforward way to drive the tests from pure python is a big plus for those of us who like that sort of thing. We can use try:finally to make sure clean-up code gets hit, and httplib2 to talk to the server to actually perform the clean-up. Control is easy in python. A quick peek at the code indicates that it does pass unrecognized options in to the test suites themselves, which means that we can code up test suites that understand additional control parameters as we need.

The Good News

While I did stumble on a couple of issues, I am very happy to report that the folks in the #windmill channel on freenode are very helpful. Even better than that, they’re very happy to get my bug reports, and are very responsive about fixing them. I spent about a day playing with Windmill a few months back. In that time I uncovered one bug and one usability weirdness. They were both fixed within days. The issues I’ve raised this time are already recorded as issues in their tracker, and I’m told that they should be resolved by the end of the week.

Conclusions

Takeaways? I’d say “cautiously wildly enthusiastic” best describes how I feel. Windmill is very close to delivering on its promise, making it ridiculously easy to generate robust, JavaScript-supporting test suites that can be trivially run on IE, Safari, and FireFox. If the pattern of developer responsiveness continues, and the issues that come up are either easily worked around or are resolved within Windmill itself, then I think this really hits the sweet spot. It didn’t take long for me to hit a couple of pretty big issues, however. If it turns out that there are a lot of similar bombs in there, that the problems stack up faster than the developers can deal with them, then it would probably end up being a headache for us.

I don’t really expect this, however. I know OSAF is using Windmill internally, and the developers that I’ve had contact with seem very enthusiastic about getting my reports and getting the problems resolved. I’m going to continue my exploration, and I hope to generate a thorough set of Windmill tests for the Plone 3-based OpenPlans stack over the next week.

This week was my first as a regular TOPP employee, after having not been here since I was an intern last summer. The current plan is for me do a few rounds of rotations, pairing with folks for a week or so to get a taste for what they’re doing and (hopefully) contributing something helpful. After getting settled in a bit, I began my first such pairing with Jeff, working on adding CAPTCHA support to the new user registration page on Trac provided by the AccountManager plugin, per ticket #8.

Broadly speaking, there are two primary ways to tackle this problem. The first is to write a stand-alone plugin that implements the IRequestFilter and ITemplateStreamFilter interfaces. The second  approach comes in two parts, the first of which is to modify the AccountManager plugin to provide a new extension point for making additions to the user registration page. The second part is to then write a plugin that implements this new interface to actually add the CAPTCHA to the form and then verify it is correct upon submission.

As an example of the first approach, I wrote a quick and dirty prototype, CaptchaAuth. To explore the second option, I wrote RegistrationConfirmationPatch to modify AccountManager to include a minor variation of the IRegistrationConfirmation interface suggested in ticket #8. I then wrote SimpleCaptcha to implement this interface and provide very basic CAPTCHA support.

As it stands, the pure plugin approach, CaptchaAuth, is a little hackish. While it does have the advantage of not requiring any modifications to AccountManager, it has some downsides. First, it relies on an XPath to grab on to the portion of the form to add the CAPTCHA to. This means that future changes to AccountManager could inadvertently break it. Second, error messages are poorly integrated, and as of now raise an ugly TracError instead of looking like other form validation errors on the page.

The RegistrationConfirmationPatch/SimpleCaptcha approach, on the other hand, has the downside of requiring modification to AccountManager, but does not have the other problems mentioned above. It also has the advantage that the patch provides a generalized interface that could be used to add any additional elements to the new user registration form. As an example, the same interface could be used for e-mail verification of new users.

I’ve e-mailed the Trac dev listserv to get to some input from the core developers and the community at large. Hopefully we’ll be able to work together to come up with a flexible and clean solution for allowing plugins to add form elements to the new user registration page.