tradeoffs of functional testing for the OpenCore software

It is desirable to support functional testing of the OpenCore software and operational testing of running instances in addition to the  current unit and integration tests that are present

Issues:

• Fragility: are the tests easily broken or fooled?
• Test Developers:  who will develop the tests?  who will use them?  are the tests targetted at software developers or users?
• Automatability:  can the tests be run with a nightly build?  is user intervention required to run the tests?
• Platform: are the tests targetted at a live site or a 'dummy' site that lives only in a testing framework?
  

Several pieces of testing software were considered:

• selenium
  

• testbrowser
  
• twill and scotch

Several tests of OpenCore were considered:

• creating a user
• user login
• starting a project
• editting the project's homepage
• logging out

• selenium:

  Selenium is a test tool for web applications aiming towards browser comptability testing and functional testing. There official selenium is developed by OpenQA, but there is also Zelenium for Zope, and PloneSelenium for Plone, which build on the Selenium framework.

  selenium saves its test files in a 3-column html table -- everything not in a 3-column row is treated as a comment. This contributes to easy readability -- see the login test for an example of the IDE output. Selenium has its own language, named Selense. Tests are available in the subversion repository. You can also Browse Selenium Tests
  

  • selenium: http://www.openqa.org/selenium/

    Selenium has three components:

    • Selenium Core: http://www.openqa.org/selenium-core/

      Selenium Core is run by opening a file, TestRunner.html, and loading a test suite, which is a one column table in html that lists all the test files. The open commands here use absolute urls. However, because of cross-site scripting (XSS) security restrictions, selenium core must be used only on local files.

    • Selenium IDE: http://www.openqa.org/selenium-ide/

      Selenium IDE is a Firefox plugin that allows recording and playback of Selenium tests.  It gets around the XSS restriction using chrome urls. The commands may be played back relative to a base URL input by the user.  It seems to work relatively well, although specification of test assertions is clunky and more technical than the rest of the gui.
      

    • Selenium Remote Control: http://www.openqa.org/selenium-rc/

      Selenium Remote Control (SRC) is a tool for automating selenium tests. Unlike Selenium Core, it is not hampered by XSS restrictions, as it uses a client-configured proxy to trick the browser. SRC has wrappers for several languages, including python. There is a Selenium Server component, which stands between the browser and the website.

      SRC does not appear to support running the HTML table tests generated by the IDE, but we were able to create a python script that supports this behavior fairly easily (SeleniumTableRunner.py)
      

  • Zelenium: http://www.zope.org/Members/tseaver/Zelenium (svn version available at svn://svn.zope.org/repos/main/Zelenium/trunk)

    Zelenium exposes a version of the web based Selenium test runner inside Zope. A Zelenium "Zuite" object can be created within the Zope hierarchy representing a  test suite.  Selenium HTML table test files are added inside the Zuite object and made available when accessing the Zuite object.  URLs which are not absolute refer relative to the URL of the Zuite object.  The HTML tests may also be augmented with Zope TAL template directives if the tests are added using Page Templates.  Using TAL allows for much more flexible tests than Selenium usually supports, but these tests are also incompatible with the rest of the Selenium products. Zelenium runs an older version of Selenium (version 0.6.0, compared to the latest=0.7.1 for Selenium core [as of Aug, 2006]) which is included in the distribution.
    

  • PloneSelenium: http://plone.org/products/ploneselenium, http://ingeniweb.sourceforge.net/Products/PloneSelenium/
    see also a presentation here

    PloneSelenium allows developers to create Selenium test stuites with a single python script. PloneSelenium requires the svn version of Zelenium. Once the tests are created, they can be run in batch mode in Zelenium.

  • Selenium Pros/Cons:

    Pro:

    cross platform works within a browser (Firefox, IE, Safari, etc) can handle javascript has a decent firefox plugin for creating a first pass at a test using a browser

    Con:

    must have a browser to run Selenium tests somewhat cumbersome to automate, requires a windowing system, external server and python scripting easily retargeting html table tests to different locations appears to be supported only in the IDE seems confused by some redirects HTML table scripts alone may be somewhat limited for complex tests

---

• zope testbrowsers

  The zope testbrowser exposes a python API for simulating a web browser. Like twill, zope testbrowser wraps mechanize. There is a useful variant of the zope.testbrowser available with the Five Product (Five.testbrowser) which exposes the same API, but communicates directly with the ad hoc zope instance created as a fixture during a ZopeTestCase. Unfortunately, the Five.testbrowser only works with doctests.  Tests are available in the subversion repositiory. You can also Browse Doctests or Browse Unittests.  Both of the testbrowsers may be run via zopectl test if you put the tests in the Products/OpenPlans/tests directory and run zopectl test -s Products.OpenPlans
  • zope.testrecorder: http://plone.org/documentation/tutorial/testing/zope-testrecorder

    Zope testrecorder is a Zope product which records a users interactions in a web browser, similar to Selenium IDE. zope.testrecorder outputs either the Selenium HTML format or python doctests which make calls to the zope testbrowser.  Examples of output is recorded at zope.testrecorder.output .  It does appear a bit buggy -- port numbers are chopped off,  the wrong modules are included, and sometimes two running instances can be seen. Tests output using this were somewhat misleading and needed to be hand edited. For some areas, such as specifiying test success cases, it was easier to use than the selenium ide.
    

  • zope.tutorial: http://svn.zope.org/zope.tutorial/trunk/

    zope.tutorial uses zope.testbrowser doctests like those output by the zope.testrecorder to create functional tutorials that can be stepped through by users.The zope.tutorial may be checked out from subversion at svn://svn.zope.org/repos/main/zope.tutorial/trunk. Minimal documentation is included, and as investigation seems to indicate, there is no website.
    

    We haven't had any luck installing zope.tutorial under Zope 2.9 yet, but we were able to get it
    going under Zope 3. zope.testbrowser installation instructions   Results were mixed at best, any errors on the server end seemed to hang the browser.
    

  • testbrowser Pros/Cons:

    Pro:

    The Five.testbrowser runs against an ad hoc instance of the project in the zope testing framework. Since there is no `live' site to deal with, things may be freely modified without having to do clean up or worry about interfereing with existing data. (though anything that can be redirected from python can also do this by calling ZopeTestCase.utils.startZServer and using the port number returned eg twill) It's all python, so there is plenty of flexibility it comes with zope some neat supporting projects are around even if they aren't very mature

    Con: Since the testbrowsers simulate a browser, rather than drive one, issues like browser compatability are not tested no javascript support (but this didn't affect things in our testing) more complex than twill to play around with outside of the zope testing framework

    
    

---

• twill 

  twill advertises itself as 'a simple scripting language for web browsing' and is capable of automated testing. Twill is written in python and based on mechanize. Twill operates on live sites and has its own limited scripting language which supports variables. An analogous python api is also present as well as the ability to load and execute twill scripts from within python. It also comes with the twill-sh, which is useful for trying out twill commands as well as finding page items (forms, etc) without having to search through the page source.  Twill scripts for our test cases and a harness to rig them into the zope testrunner can be found in the subversion repository, you can also (browse)
  

  • scotch: http://darcs.idyll.org/~t/projects/scotch/doc/
    scotch is 'a collection of somewhat arcane WSGI modules' according to its site. (`What is WSGI?')  scotch supports recording http traffic to disc which may be played back using a proxy.  Other similar arrangements are listed on the scotch website. Of particular interest is the utility which allows conversion of traffic captures to twill scripts. Unfortunately, the output was extremely verbose and, like most recorders, did not work out of the box.  At least for the simple tests we were performing, it was a little more trouble than it was worth. instructions for capturing a twill script using scotch  
    

  
  

  • twill Pros/Cons:

    Pro:

    conscise programmer readable domain specific language extensible / tweakable via python easy shell for experimenting not hard to automate, rigging under zope testrunner was fairly simple retargetting is fairly trivial, pointing at zope fixture under testrunner was not difficult python api

    Con: Since the testbrowsers simulate a browser, rather than drive one, issues like browser compatability are not tested no javascript support (but this didn't affect things in our testing)

Other Links:

• Our Experiments -- (Browse)
• Functional and system tests with zope.testbrowser
• Beautiful Soup, a Python HTML/XML parser