OpenCore Software

This showed up on my radar today. It’s a cute little hack, if a bit rough. (I’ve already posted a couple patches).

It could easily be generalized, there’s nothing really zope-specific about it, all it really cares about is a path to a common-apache-format access log and the PID of a process to watch memory. The result is a nice little Flot graph where you can select a range and see what URLs were hit during that time.

Here’s a screenshot of an example from a flunc run against opencore (the “basic” suite):

So what does this tell us really? I’m not sure. Multiple runs show that the graph generally starts around 400 MB and ends around 500 MB, but what happens between does not seem consistent. So I’m not sure if this little app is really that useful for opencore.

One of the things that’s plagued our buildbot config for a while is how to know when the web servers are really up and ready to take requests. For a while, I had buildbot configured to just sleep for a while and then start running flunc. This would eventually fail when the box was under heavy load; the flunc tests would start when things weren’t ready . I suppose I could keep doubling the sleep time, but I wanted something more meaningful.

I finally came up with a hack that seems to be reliable - use wget to actually request a page from the app and block until it returns or times out. Something like this:

fac.addStep(ShellCommand,
            command=('%s/bin/supervisord && sleep 10 && wget '
                     '--retry-connrefused --tries=10 -T 100 --spider '
                     'http://localhost:%d' % (OCBASEDIR, ports[1])),
            description=['start services', 'for functional tests'],
            haltOnFailure=True)

The full config (or rather, a template for it) can be seen in our source control here.

Another problem was that the services sometimes didn’t get shut down properly. I hacked a bit on jeff’s very handy portutils package until it seemed to do the trick.

There have been complaints — valid complaints — about difficulties dealing with our stack, and debugging errors in the system.  The response I would offer is that we need to debug the debuggabilty of these systems.  We could try to change architecture and keep our code more monolithic so everything shows up in a single stack trace, but I see that as a dangerous kind of overreaction; a panicky response where we overestimate the problems we know about and underestimate the problems we don’t know about.

If it is difficult to deal with the stack, then we should fix those problems.  If it’s too hard to debug, that is itself a bug.

Here are some general things I think we could do to improve our system:

1. More ubiquitous use of logging (with logging).
2. Aggregate log viewers, so we can see all the messages that are generated by any part of the system.  Because there are sometimes side effects in other parts of our stack outside of the application you are interacting with (especially with the introduction of Cabochon) we need see everything that is going on.
3. Soft failures, well logged.  Also more warnings.  Everyone is tired of all the warnings that come out of Zope, but that’s just because it’s not our code that is causing the warnings.
4. More assertions in code.  Exceptions should be raised as soon as possible.  Type assertions are okay, especially for things like strings and numbers where there aren’t generally other types you could usefully use to replace them.  Generally immutable objects without substantial behavior, like strings, integers, in some contexts lists, tuples, and dicts, do not benefit from polymorphism and sometimes potential polymorphism is just a bug waiting to happen.  (We shouldn’t put in checks everywhere, but if a bug occurs we should put in a check even if the actual bug is elsewhere in the system.)
   
5. Error messages should be helpful and include sufficient information to understand the error.  Bad or unhelpful error messages are a bug.  So if you ever get an exception from some code like “assert x is None”, you should change it to “assert x is None, (’x should be None (not %r)’ % x)”.
   
6. Centralize error reporting, so that errors in any part of the system are consistently presented.
7. Up-front checking of configuration parameters for general sanity.  Any configuration error that we make is something we should check for.  It doesn’t matter how stupid or misinformed the configuration mistake seems, if someone made a mistake we should add a check for that mistake.
   
8. Greater use of code analysis tools (like PyChecker, pyflakes, jslint) and putting together the scripts to make this checking easy (perhaps part of the fassembler builds).  These can’t find lots of bugs, but they will usually find some bugs.  And of course every valid bug found is one less bug.
   
9. Make more use of documentation tools, and document the core parts of our system that people need to understand.  Apydia looks promising, though Epydoc would be fine too (and it’s older and more mature).
   
10. When bugs are hard to debug, make a ticket for it.  For example, does fassembler fail in a weird way?  Even if you figure out why and fix it (e.g., some weird system dependency), ticket it up anyway.  It probably will always fail, but it should fail in an unweird way.
    
11. When you are debugging code, try to put in debugging code that you can leave in.  For instance, use well-structured logging messages instead of print statements or pdb.
12. Make sure all objects have useful __repr__ methods.  Include a little extra code in __repr__ to show the interesting structure of the object, so that the data doesn’t become overwhelming (e.g., sometimes leaving out attribute values if the value is None).
    
13. If you encounter an error like an attribute being set to the wrong kind of value, consider replacing it with a property with assertions.  Automated testing is nice, but runtime checks with simple tests that exercise functionality are better.  Runtime checks happen during simple exercise tests, and also in actual production code, and potentially during other tests that you might not have realized exercise that code.
14. When writing new code avoid being needlessly permissive.  For instance, if you are retrieving a value from a dictionary you can use a_dict[key] or a_dict.get(key).  If you aren’t sure which one to use, use a_dict[key] (that’s the more attractive syntax for a reason).  Of course if there’s actually a reason to use a_dict.get(key) then use that.
    
15. If you ever put in a hack, always always always put in a comment describing why it’s there and what its purpose is.  Later people should know that there’s something fishy there, so they can think harder while reading that particular code.
    

If we make these improvements in the stack and adhere to these rules while writing and maintaining code I think we’ll all have a much more fulfilling development experience.

To make for a smoother deployment next time, I’ve thought of a few things:

• we need to write more tests and we need to run them; when I joined the firefighting of the 0.9.7.7 deployment, the unit and flunc tests were broken in several spots and I suspect they were broken when the site was deployed; also, there were some things like the password request form that had a runtime error and would have been easily caught had there been a test for it
• we need to remember to merge fixes between svn release branches and the trunk; there was at least one bug that was fixed in the trunk but not merged into the release branch and I believe there were some things that were fixed in the branch but not merged to the trunk; this can get complicated because when you fix a bug you need to fix it on both the branch and trunk and be able to run the tests on both to make sure you haven’t broken anything
• when using trac, I found it useful to specify what site the bug was found in and what release of opencore was currently running on the site; after fixing it, I would record both the trunk and branch rev number that I made and then post another entry when it was deployed and verified on the live site and only then would I close it; this last step might only be appropriate during “firefighting” mode
• finally, I think when we are firefighting, having frequent standing meetings, as we did do, is helpful

Just landed a few changes that had been hanging out in the wings for awhile that I wanted to let folks know about. This has to do with getting flunc into the opencore distribution, a personal itch stemming from having extra steps to set up the ftesting (which I found myself plain forgetting to do).

So here’s where things are::

1. opencore’s ftests now ship with opencore (in… opencore/ftests). This should help keep things in sync. For bbb purposes, the ftesting bundle is updated to look at opencore trunk’s ftests.
   
2. flunc now comes with a setuptools extension that allows for the running of flunc tests inside a package from setup.py. This doesn’t provide overwhelming convenience, but does take the guess work out of remember to use -p. Later, this hopefully will grow a bit more sophisticated as a test finder. For now the following command will run flunc (provided your zope is running on localhost:8080). It will also interpret the normal flunc options.

     $ cd opencore
     $ python setup.py flunc -vw

3. flunc is now a dependency for opencore, meaning the flunc test running script is installed everytime you install opencore. And it works the same way as always, so if you choose, you can run the tests like so:

   ­

   $ flunc -vwp opencore/ftests all

   ­

As to which form of invocation works better, it’s apples and oranges (depending on perhaps what you are doing).  Framework-wise, there are some interesting things the setuptools command could let us do with organizing, finding and running ftests without changing the simplicity in how flunc works as a standalone script. More importantly, whichever way you run it, you are running the tests in the same way (or explicitly deviating by using the spelling in #3).