Greetings from Testlandia

I've spent most of this week working on cleaning up our testing situation.  Earlier in the week I was working on our zopectl tests; today I've been playing more with functional tests.  I've given some attention to our ailing (and failing) flunc tests, and I also took a bit of time to play with windmill.

Flunc

First, our flunc tests.  The situation there wasn't as bad as I had feared.  For the last couple of weeks, these tests have been MOSTLY running for me, only to fail during the home page tests in the project_homepageredirect suite.  I thought that the problem was due to the flunc tests not reflecting changes in the code.  After I took a closer look, though, I realized that the tests were right, and they were really indicative of a bug in our software.  Imagine!

That's right; turns out our home page selection stuff wasn't working at all.  Ya see, whenever a new project is created, it automatically creates a project-home wiki page and then calls "project.setDefaultPage('project-home')".  setDefaultPage is pretty straightforward; it tells Zope that when the viewer lands on the project with no add'l path information, the 'project-home' page is what should be displayed.

Then we also had a custom view on all project objects, which checks what the user-set home page should be, and redirects the user to that page.  Problem is, the setDefaultPage setting was trumping this; our custom view was never invoked, because we told Zope we wanted 'project-home' to be the default page, full stop.

I've removed the setDefaultPage call.  That of course broke a bunch of getDefaultPage calls elsewhere; I changed those to just grabbing the project-home page by name.  Not very elegant, and needs to be revisited, but I was running out of time and needed to get the tests passing again.

Windmill

On another front, I started playing around with the OSAF's windmill product.  Windmill is more like Selenium than it is like twill; i.e. it works by driving a browser to talk to your site.  It works by starting a server process through which it proxies access to the actual site, and from which the browser is managed and introspected.  It's got a nice IDE for running and generating tests.  And, unlike Selenium, it is very scriptable; windmill has a command line client, an interactive python shell, and tests can be saved either as JSON or pure python code.  It does require a browser to run, however; this isn't something you could have a server run automatically.

In order to get a feel for windmill, I started by trying to port the very first flunc test we have over to windmill.  In the process I hit a few usability bugs; things in the IDE didn't work like I expected them to.  In fact, I couldn't figure out what i was supposed to do.  So I popped over into the #windmill channel to ask a few questions.  The folks there were VERY eager to help folks who were using windmill; one of the devs said they were in "Get a bunch of users" mode.  They answered all of my questions in a very short time, and opened issues in their trac for everything that came up.  In fact, they closed two of the usability issues I stumbled across within the hour.  Very nice.  :-).

So, with a little bit of help, I was able to get a passing windmill test.  Here's what it looks like in python:

­

# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient

def test():
    client = WindmillTestClient(__name__)

    client.click(link=u'Create account')
    client.type(text=u't', id=u'id')
    client.type(text=u'test_error 123$!@$#%!#@$%', id=u'id')
    client.type(text=u'test name 12345%@!#$', id=u'fullname')
    client.type(text=u'bad email address', id=u'email')
    client.type(text=u'a', id=u'password')
    client.type(text=u'b', id=u'confirm_password')
    client.click(name=u'task|join')
    client.asserts.assertText(validator=u'The login name you selected is already in use or is not valid. Please choose another.', id=u'oc-id-error')
    client.asserts.assertText(xpath=u"/html/body/div/div/div/div[2]/div/form/fieldset/table/tbody/tr[2]/td[2]/span", validator=u'(optional)')
    client.asserts.assertText(validator=u'That email address is invalid.', id=u'oc-email-error')
    client.asserts.assertText(validator=u'Passwords must contain at least 5 characters.', id=u'oc-password-error')
    client.click(name=u'task|join')
    client.asserts.assertText(validator=u'Please enter a password', id=u'oc-password-error')

Pretty straightforward.  I run this from a command line using "windmill firefox exit http://localhost:8080/openplans tests=create_user_with_errors.py".  This launches firefox and the windmill console, loads up the suite, and steps through the actions.  You can see what's going on in the browser in front of you as it works.

I generated this test using the IDE.  Basically, you launch a windmill browser session, then click 'record' and start using the site.  When you want to check something, you click on the assertion button and then click on some element of the page; windmill will either assert that the node exists, or, if the node is a span or a div tag, it will assert that the node exists and that the text within the node is what is currently on the page.  Very nice.

Windmill is definitely still a little rough around the edges.  But, according to the devs I spoke to, it's getting serious use within OSAF, and they love love love it.  I can see why.  I think we could, using windmill, get to a point where we're generating our functional tests at the same time we're doing our initial browser QA testing.  Once the tests are generated, they're just python; we can push them around and invoke them however we want to.

There's more I want to cover, but it's 7:15 on Friday night and I need to get out of here.... more soon...