This was the task dependency diagram for yesterday evening. So, so long ago, Rollie was talking about task tracker design, and I didn’t let my being uninformed stop me from being opinionated. But for the record, my dream task tracker would do/look something like this. Sorry that it’s too big for Wordpress:
(click to enlarge)
In his blog post, cabraham takes a stab at how we define success by saying:
Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.
My concern here is similar to my previously stated concerns that we are pulling each other in different and incompatible directions. TOPP is a large organization with a variety of projects under its wing, making it difficult to steer projects such that they complement each other rather than conflict. Using the same metrics for success for both NYCstreets.org and OpenPlans.org is an example of this.
While there has never been a clear definition as to what OpenPlans is or who it targets, one shared vision of OpenPlans was as a website geared at empowering citizens to organize and effect change, specifically in regards to issues of urban and community planning. This vision dovetailed nicely with the goals of our advocacy branch—as Streetsblog and Streetfilms developed the propaganda, OpenPlans.org would provide the tools. Yet OpenPlans.org was conceived of much too broadly to ever get very far in developing a community to complement those on Streetsblog and Streetfilms. As a result, NYCstreets.org was launched to combine the tools provided by OpenPlans with the focus of purpose that drive Streetsblog and Streetsfilms.
As a consequence of the launch of NYCstreets.org, however, OpenPlans.org was stripped of a large part of its stated audience: no longer can OpenPlans.org build a community around urban planning issues without directly competing against its sibling, NYCstreets.org. Direct competition between the two sites hurts both and benefits neither.
There has always been confusion around the distinction between OpenPlans, the software, and OpenPlans.org, the website, and many have viewed them as two sides of the same coin. Although narrowly conceived, there was a never a problem with this view and we muddled along just fine. But now that we are running a service—NYCstreets.org—using the OpenPlans software stack we have an opportunity to reposition ourselves along the lines of the GeoServer model: we must reconsider OpenPlans as a product, capable of being easily installed and customized, rather than a tailor-made package designed solely to run our OpenPlans.org service and no more. The definition of success cabraham proposes is reasonable for measuring the success of OpenPlans.org, but to continue thinking of OpenPlans as having a one-to-one relationship with OpenPlans.org is short-sighted and will likely result in failure.
The possibilities of success are much richer for OpenPlans if conceived of as a product. We are no longer limited to the constraints of a single website with a single goal but instead have the opportunity to build software that can address many problems—ranging from facilitating Plone sprints as OpenPlans.org currently does, to organizing communities in the form of NYCstreets.org, or even facilitating the project management needs of small local governments. OpenPlans as a product can’t be over-designed and over-engineered to meet overly specific use cases, but should instead be built to be flexible, easy to use, and broadly useful.
Despite having an opportunity to broaden our scope, we must be careful to focus our efforts on improving the usability of our product. NYCstreets.org provides us the opportunity to explore real-world users and situations around which to build our software, and the existing OpenPlans.org website will continue to provide an entirely different set of examples. My suspicion is that we will learn that our model of segregating functionality by type (separating pages from tasks, for example) is not as useful as the BackPack model of allowing multiple content types on each page—after all, users don’t care about what type of data something is as much as the context to which it belongs.
It is my hope that OpenPlans will mature to the point that it can consider sustainability models like those proposed by Cholmes for GeoServer, perhaps even becoming as ubiquitous for project management as WordPress is for blogging.
I’ve been thinking a bit about Nick’s question, “how do we define success here at TOPP?” I’d been thinking about this for some time since coming up with TOPP News.
It seems to me that organizations all have some mission, or purpose, “why they exist.” They also define themselves by a set of shared “values”, things that the organization believes to be worth holding on to. Defining success at TOPP then would be defining how well we are achieving our mission and how well we are holding on to our values. The tough part is usually how to measure these.
So first there’s our mission: to build technology to enhance the role of the citizen in democratic society.
This is being done through StreetFilms and Streetsblog, by bringing people’s awareness to the quality of life on the streets of NYC and by educating people in how to improve street life. Success here would be a measure of how well we are achieving this by measuring, say, the overall traffic of the sites and the quality and quantity of people’s comments.
Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.
And then GeoServer which is an open, interoperable infrastructure of geographic information. I’m not as familiar with the current uses of GeoServer but I would imagine that the success of GeoServer could be measured by looking at the quantity and quality of real-world projects that use GeoServer to accurately inform their real-world decisions.
Part of accomplishing our mission is to be continually looking for and trying out new ideas and projects that use technology to enhance the role of the citizen (eg. Melkjug).
Next, we need to look at our values. These are things such as:
Having defined both our mission and our values, I would therefore conclude by defining success at TOPP as: “achieving our mission while also acting in alignment with our values.”
So I was going to work on some issues in listen. First thing I did was run the listen tests in a fresh opencore build:
$ zopectl test -s Products.listen ... Total: 60 tests, 0 failures, 0 errors
So far so good. I made a few edits to explore the issues and ran again.
$ zopectl test -s Products.listen
...
Set up Products.PloneTestCase.layer.PloneSite
Traceback (most recent call last):
File "/home/pw/opencore-pristine/opencore/src/Zope-2.9-branch/bin/test.py", line 125, in ?
sys.exit(testrunner.run(defaults))
File "/home/pw/opencore-pristine/opencore/src/Zope-2.9-branch/lib/python/zope/testing/testrunner.py", line 271, in run
failed = not run_with_options(options)
File "/home/pw/opencore-pristine/opencore/src/Zope-2.9-branch/lib/python/zope/testing/testrunner.py", line 433, in run_with_options
setup_layers, failures, errors)
File "/home/pw/opencore-pristine/opencore/src/Zope-2.9-branch/lib/python/zope/testing/testrunner.py", line 604, in run_layer
setup_layer(layer, setup_layers)
File "/home/pw/opencore-pristine/opencore/src/Zope-2.9-branch/lib/python/zope/testing/testrunner.py", line 692, in setup_layer
layer.setUp()
File "/home/pw/opencore-pristine/opencore/zope/Products/PloneTestCase/layer.py", line 28, in setUp
setup.deferredSetup()
File "/home/pw/opencore-pristine/opencore/zope/Products/PloneTestCase/setup.py", line 315, in deferredSetup
SiteSetup(*site).run()
File "/home/pw/opencore-pristine/opencore/zope/Products/PloneTestCase/setup.py", line 162, in run
self._setupRegistries()
File "/home/pw/opencore-pristine/opencore/zope/Products/PloneTestCase/setup.py", line 216, in _setupRegistries
portal = getattr(self.app, self.id)
AttributeError: plone
Whoops, I broke something in my edits. Right?
Well, no. After much tearing of hair, I noticed that I could reliably do this in a totally fresh build:
After tearing some hair, I discovered in the debugger that somehow, PloneTestCase.setup is broken on the second run. The plone site is created, transaction.commit() is called, at which point the newly comitted site vanishes in a puff of smoke.
WTF.
After yet more tearing of hair, I discovered that I could make the tests run again by doing:
find src/Zope-2.9-branch/lib/python -name “*pyc” -exec rm -f {} \;
And of course, the tests only work for one run after that. On the next run, kaboom. Until I remove the bytecode files again.
After playing human-binary-search for a while, I narrowed it down to a single file which has no obvious relationship to anything persistent. Apparently I can do this reliably:
$ zopectl test -s Products.listen ... AttributeError: plone $ rm -f Zope-2.9-branch/lib/python/zope/app/form/browser/itemswidgets.pyc $ zopectl test -s Products.listen ... Total: 60 tests, 0 failures, 0 errors $ zopectl test -s Products.listen ... AttributeError: plone $ rm -f Zope-2.9-branch/lib/python/zope/app/form/browser/itemswidgets.pyc $ zopectl test -s Products.listen ... Total: 60 tests, 0 failures, 0 errors
Weird, right?
I wondered if it was an issue with something imported by itemswidgets.py. There aren’t many imports in there, and after removing bytecode for all of them, I gradually narrowed it down to one import: xml.sax in the standard library. I shit you not. There is no way I can think of that xml.sax could impact PloneTestCase’s demostorage setup. But it does:
$ zopectl test -s Products.listen ... AttributeError: plone $ rm -f /usr/lib/python/xml/sax/*py[oc] $ zopectl test -s Products.listen ... Total: 60 tests, 0 failures, 0 errors
Of course, Rob Miller tried to reproduce and he never sees this error at all.
So, I’ve been here at TOPP for a bit more than two months now and my contributions as far as fuzzy, non-software-related stuff like blogs have been pretty close to nil. I’m not sure that’s a particularly bad thing, but nonetheless a blog post or two seems unlikely to go awry, so here goes.
The conversation that prompted me to even consider posting here occurred this past Monday at the impromptu Geoserver front- and back-end teams standing meeting that we held after Nick informed us that the OpenCore folks could have a better meeting without us. Seb pointed out that what I’ve been working on for the past few weeks is pretty opaque to him and that it might be worthwhile to blog a bit about it. Given the success of Seb’s blog posts, I imagine he’s right about that, but I’m not going to blog about it now. If you’re really interested, there’s some info about it on the Geoserver wiki. (http://docs.codehaus.org/display/GEOSDOC/RESTful+Configuration+API and http://docs.codehaus.org/display/GEOSDOC/RESTful+User+Management+API)
Since that conversation with Seb, I’ve been thinking a bit about what exactly my role at TOPP is, in part due to conversations with Lily, who has been asking me, and presumably the rest of us code monkeys, what it is that we do and how we refer to ourselves, etc. When she asked me I was pretty firm about being a software developer as opposed to a programmer or software engineer, even though officially my title is Software Engineer. I think what I had in mind was kind of a strict taxonomy of byte wranglers (note that I’m using Capital Letters to indicate these are my own definitions and not anything particularly formal):
* A Programmer sits around, takes some very literal, low-level (and, by my subconscious association, uninteresting) specifications and translates them from English to computer-ese. This is what I think of when I use the term ‘code monkey’ in a somewhat serious context, someone that basically sees the process of creating software as a simple conversion rather than a creative process.
* A Software Engineer takes higher-level requirements in the form of detailed specification documents (or, in the absence of such documentation, creates it) and converts them to the kind of specifications that a Programmer can use to make an actual piece of software. Software Engineers treat the process of creating software as a scientific phenomenon with a sort of physics of its own, measuring things like Effective Lines of Code and Defects Introduced Per Defect Addressed and trying to infer useful information from them.
* A Software Developer takes higher-level, but not necessarily very specific requirements and makes them into software. For the Software Developer, the process of creating software is an incremental one that involves constantly referring to the people that are going to be using the software, or at least a working knowledge of the problems that the software is trying to solve. Software Developers treat the process of creating software as a solution to a problem, and define success therein by measuring how well the problem is solved rather than software metrics.
While writing these descriptions out I noticed that I’ve defined Programmers and Software Engineers by what they do and Software Developers by how they think about things. It doesn’t really seem like the three are contradictory, and I know that what I’ve been doing over the past 2 1/2 months has fit each of these roles (and probably each combination of them) at different times.
So I suppose I’ve not really said much, but then I always feel that way after talking about me. At any rate, feel free to refer to me as a Software Engineer or a Software Developer or a Grand Vice Poohbah of Java-Based Endeavors; it seems like the definitions are equally nebulous for all of them.
PS: As I was writing this it occurred to me to consult the oracle wikipedia, and I found that the page for Software Developer redirects to the one for Software Engineer. Shows what I know, I guess.
Nick brought up some very good questions in his “Year in Review” post—none of which I intend to answer here.
Instead, I want to address a larger question he implicitly poses: how do we define success? At the moment we simply ignore the question altogether and are content to ride the wave of our day-to-day inertia, but it is a question that is crucial to our existence as an organization. No doubt we each have our own conception of what would make us successful: for some it is the quality of our software at the code level, for others the value it brings to our users, and for others still a sense of how ‘cool’ it makes them feel. Yet, without a shared vision of how we define success and the path to get there we will continue to be pull each other in different, often incompatible, directions.
Conventional measures of success like market share and profitability, however convenient and problematic they may be, simply don’t apply to a non-profit like ours. This puts us in the difficult position of having to define our own success. As Cholmes is so fond of saying, we’re not a non-profit but a ‘for-benefit’, so do we define success as the amount of ‘benefit’ we produce? How would you go about defining our success?
Coming up:
I didn’t really participate in bug day, as my difficulty reproducing things gets in the way. So I kept working on fassembler. After struggling a little with opencore, I decided to do the wordpress build, which was pretty straight forward. Then I just didn’t feel like struggling with opencore again.
When I don’t want to do what I probably should do (top priority stuff), I’ve found a good way to cope is to do stuff that I know needs to be done someday; typically little packaging things. In this case I added docstrings to everything in fassembler, and reviewed the FIXME’s that had accumulated. I’d consider this kind of work “puttering”. I really need to spend some time doing this for Deliverance; though really it would have been better if I had done it when Deliverance was more at the front of my brain; it’s kind of pushed towards the back now.
But Fassembler is mostly puttered out. Now I have nothing to do but deal with that opencore build 
So, as 2007 draws to a close (I know it’s only November, but I’ve already bought all my christmas presents so I’m feeling a little ahead of schedule), it seems appropriate that we take a moment to look back on what we’ve accomplished this year, and think about where we’re headed for 2008.
Rather than try to sum up a year in my own words, I thought I’d put it out to you. Here are a few questions to get you started (relating to OpenCore/OpenPlans and TOPP):
I’ve been talking about doing this for some time, so I decided to go ahead and try it to see how it would go. Ra, Whit, Josh: I’m hoping to get feedback from you guys. Josh, I know that you spent some time on how the current context menu gets generated. Let me know if you think this is a step forward, or a step backwards.
The results are here: http://trac.openplans.org/openplans/log/opencore/branches/viewlet_menuitems
The second commit in the branch uses more mixins to attempt to cut down on code duplication, even though I think there’s actually more code there. I struggled to find a balance between removing duplication and over-engineering the thing.
Anyway, here are my thoughts about it:
It turns out that viewlets simplified things a lot less than I thought they would. The reason why is that I was unable to take advantage of the “for” attribute as much as I hoped, which is where I thought most of the savings would have come from. Here’s the problem:
We need to know if we’re in the “context” of a project, member area, searching, or no context (portal). This precludes me from registering viewlets for an IProject, because what I really want is whether the context object, or any of its parents, provides a particular interface. I essentially want the interfaces that are provided to be “acquired”. Ideally, there would be a zope way to specify this. I ended up having to do all these parent context checks myself, instead of having zope take care of them.
This is not the first time this problem has come up, and I don’t think it will be the last. When this came up last was when we wanted to only give xinha to experimental projects. So, we marked projects as IAmExperimental, and figured this would take care of all our problems. However, we are viewing pages within a project, and cannot lean on zope’s machinery to use a different registration based on the project marking. So, we added the experimental check in code. The view class itself checked if its parent provided IAmExperimental. It was simple to get around this one case, but as we roll out more experimental features, this will become more difficult to manage.
Something I considered to get around this, is simply marking all objects with an interface as they are being created, with their particular context. Simply register an event handler to listen to when an object gets added to a container, and mark it accordingly. Then, we really can leverage the “for” part of the configuration. The code becomes much, much simpler, with almost all of it being controlled through configuration. For example, if a new page is created in a project, an event gets fired. Somebody (hopefully the project, although I haven’t look at these kinds of events so I’m not sure offhand) handles this event by marking the object with something like: IHaveAProjectDaddy. Or, more realistically, something like IProjectSpace.(Those of you who have paired with me know I like to come up clever little sentences for interface names. Clever imho of course 
We could also do the same kind of thing with the experimental projects. We’d fire the same event, and have a handler that would first check if the project is experimental, and if so, mark the new object with the experimental interface. Then we gain all of zope’s configuration power.
Somehow though, this seems like a big change. Even though it won’t be much code, it feels big. Like I’m using the wrong tool for the job. And, it’s not going to help in the case of our featurelets that don’t live in zope. These guys don’t have any zope objects to mark, so we would still have to do the context dance in this case.
As a side note, I only ran into one zope hiccup when moving to viewlets. If it’s not already common knowledge, the viewlets for a particular manager are in no particular order. All is not lost however. There is a viewlet manager sort method that you can override to impose your own ordering. I used the strategy Philipp does in the zope 3 book: sort them by name, and prefix the viewlet names with an integer. I ran into problems specifying a custom viewlet manager class. I tried using a completely empty subclass of the base viewlet manager zope uses, and still had the problem. It was a security error. Well, since zope was fighting dirty, I also had a trick up my sleeve. Since the default sort behavior sorts the viewlet objects themselves, I just added a custom __cmp__ to a base viewlet class that would compare their names. This turned out to work just fine. But, if anybody knows how to get this working with the viewlet manager (since it makes more sense to put this logic there), please let me know. Maybe there’s some five declarations I need to sneak in somewhere.
Besides all these downsides that I mentioned above, things went really smoothly. Surprisingly so in fact. And even though I didn’t gain as much as I had hoped for with viewlets, imho there’s still a gain. I got some bang out of the permission configuration, and I was able to use “for” configuration to filter out the menu items for the searches, since those don’t have child objects to deal with. This also moves us more in the direction of having component logic live with the component. The menu items for featurelets now live with the featurelet. You don’t need a part of the application to know about everything else.
What it comes down to though really, is whether the viewlet approach is easier to maintain. I think it is, and like it because it’s following a zope pattern. But, I’m biased, so I’ll let you guys be the judges of that.
Sorry I haven’t been very attentive to stuff lately. First I got sick and then Emily’s sister just had a baby, and I’ve been running around helping with all that stuff. But that’s all calmed down now.
Fassembler just started working with most of the core components. I rewrote openplans_hooks.py, which was evil, to be totally config-driven, and have supervisor managing all the processes. So when you start ENV/bin/supervisord, all the subprocesses get started up, and you can control them with commands like “ENV/bin/supervisorctl restart all”. The ad hoc start/stop situation has been driving me nuts, so I’m looking forward to this. Deliverance and TaskTracker are building pretty much exactly how I want them to. OpenCore is the big thing to tackle now, as it’s at the heart of a working site. There’s a bunch of other details — WordPress needs to be built, and its database setup made more robust, and several products need to be updated to use the central config file. It would be nice to start writing backup scripts and other data management scripts as well, like “clear all data” scripts. But I don’t see any need to do those right away, or even all at once — we can just keep adding useful scripts as they come up.
This would probably feel less overwhelming if I wasn’t constantly moving between various deeply nested directories — ~/src/fassembler/ENV/supervisor/src/supervisor/src/supervisor/supervisorctl.py is a good example. I often deal with long paths, but this seems worse. There’s a mental burden to that sort of path. But I don’t really see an alternative. At least since things are all checkouts I can handle them in-place instead of fiddling around when I need to make an edit.