HTTPMock; Build strategies (repoze.project)

HTTPMock

I've been working on a PAS plug-in that will allow cross-site authentication between OpenCore instances, so we can allow folks to use their openplans.org account names when logging in to nycstreets.org.  The OpenCore instances communicate with each other via HTTP, which adds a bit of a wrinkle to writing automated unit tests, since we don't actually want to require a running remote auth server somewhere just to run them.

I was pleased to find that we already had a nice HTTPMock object in our opencore.testing.minimock module.  It simulates an httplib2.Http object, always returning a 200 server response and content that says "Mock request succeeded!".  If you use getUtility(IHTTPClient) instead of directly importing from httplib2, then you have the option of including the MockHTTP layer in your tests to be able to fake HTTP requests.

This is a great start, but in my case I needed to control the content that the "remote" HTTP server was returning, to emulate authentication success and the return of remote user information.  I added a response content queue to our HTTPMock class, along with an 'add_to_response_content' classmethod that lets you add responses to the queue.  If there's nothing in the queue, the response content defaults to the original success message.  No major engineering here, just a handy test rig that folks should know about so they can use it when appropriate.

To Build or... well, I guess To Build is our only option, isn't it...?

There has been a lot of noise in the TOPPosphere lately about our build system, its problems, and the problems with how its development has played out.  First, let me say that I'm sorry for the personal alienation that's been expressed.  I'm sure I've contributed to some of the frustration, and that of course has not been my intent.  I've never considered the build code to be owned by just one person, however.  In fact, I actually LIKE working on our build system, and have for some time wanted to find some cycles to contribute more than just opinions to its development.  I've not found that time, however, and neither has anyone else, so we are where we are.  Also, to echo Whit's comments, I feel like our build system thus far has absolutely been a success.  I use it all the time.  It almost always does exactly what I want it to, which is construct for me a working system so that I don't have to worry about doing so by hand.  I'm very grateful for the work that's gone into it, and I'm very grateful for all of the time that it's saved me.

That being said, Ian has expressed a desire to start over, because (I hope I'm characterizing this accurately) he feels that buildit maybe adds too much complexity (or at least it doesn't fit his brain very well), and that perhaps buildit's strengths are not well matched to the needs we have of a deployment system.

Interestingly, as I was playing around w/ repoze, I realized that it has an installation mechanism that works very much like we want ours to work.  In fact, the end result (a virtualenv that is populated w/ all of the packages and configuration required to run the process) is very similar to that of the topp.build.* suite.  Usage is very simple; you easy_install repoze.project, and then you call 'repozeproject' to build the environment.

How does it work?  repozeproject takes an egg as an argument.  It sets up a virtualenv, then installs the egg, and then looks for a 'repoze.project' entry point within the newly installed egg.  If it finds such an entry point (and you haven't suppressed the behaviour with a command line flag), then it fires the entry point, which is responsible for finishing up the configuration.  setuptools handles the dependencies, and each package encapsulates its own configuration information.  Currently, the entry points are usually just using boilerplate code to create directory structures and populate them with config files, doing simple string substitution to inject dynamic values into the config.  (This would probably make a bit more sense if you took a look at what they've done for repoze.plone.)

I think this solution is simple and elegant, and fits very well with what we're trying to do.  One thing that's been a bit of a blocker for us using a similar approach is that not all of the code that we use has been packaged up as eggs.  They've solved that problem by packaging their own.  Indeed, we could take the entire 'openplans-plone25-bundle' and make that a single egg, installed into the Products namespace, as they've done with the ploneproducts egg for Plone 3.  It's also true that not all of the applications that we wish to deploy are written in python.  I don't think this is a problem, it just means that the installation egg will have few (if any) setuptools dependencies, with nearly all of the work being done by the repoze.project entry point that the egg provides.  This egg could pull in buildit (or any other build management toolkit that we may find or create) as a library, or even just delegate to a shell script that does all the hard work.

The devil is in the details, of course, but this seems like a great approach, and a great way for us to start.  It certainly feels much nicer to me than a certain other approach to managing deployment environments.