OpenCore Software

A lot of times at the Snow Sprint I was struck by a feeling that I was in the middle of a group of people who were really doing open source right. So on the plane home I started thinking about what that meant and why I haven’t felt that way at TOPP.

Well, a lot of us have said at various times that our attempts at doing open source development have often been frustrating and rarely seem to be worth the effort. Coming out of the sprint, I think that’s because we are missing a very big piece of the “True Spirit of Open Source”. Doing open source properly is not just a matter of putting the GPL on our code—it is not about making a legal contract but about making a social one. It is about committing to reuse and collaboration both internally and externally and it is well worth the effort.

Writing, finding, and using software all take time and care and with someone paying for our time and care we ought to think of these acts as investments. We should make the most of those investments by seeking code reuse and encouraging collaboration. The more we encourage heavy use of software and heavy reuse of code, the better our ability to improve and solidify it. Whether or not we’ve read Dreaming in Code (though everyone here should—as soon as possible—seriously) we all know that software is hard to do well. It’s full of unexpected behavior, prone to breaking when you touch it, and rarely optimized. We can improve it only by testing and exercising it: give it to people to use, poke it in new ways, put it under stress, see what breaks, fix it and don’t let it break in the same way again.

Laziness is a virtue

Because software is expensive to write, it is most responsible of us to write as little as we can. And because software is hard to write, it’s rarely done once it’s written; most software has tremendous room, and often necessity, for improvement. So our goal should be to maximize the impact of every improvement we make. This primarily means seeking to increase the breadth of our code’s effect: convincing other people to use our software, sharing code across multiple products, and reusing products in as many ways as we can across everything we develop.

To give a specific, if vacuous, example, we have five or more different pieces of code deployed on OpenPlans that generate and send email to users: Listen sends mail, Twirlip sends mail, WordPress sends mail, and Opencore sends mail in at least two different ways. Suppose, instead, that we had a single shared way to send mail to users. Aside from the immediate benefit of less overall knowledge to store, the different mail-sending needs of all these applications would force us to develop a stable and reliable method for sending mail, and when heavy usage of one of those dependent applications inevitably reveals an inefficiency or bug in the mail-sending code, multiple otherwise unconnected parts of our site would instantly benefit from our resulting improvements. Put that same mail-sending code on two or more websites and encourage other developers to deploy it in their applications, and we can tremendously increase both the number and variety of stress points and the magnitude of impact of each strengthening, both the opportunity and the payoff for improvements.

So maximizing adoption of every piece of software by both users and developers is a forward-thinking and cost-effective stance. It gives programmers the most opportunities to strengthen their (buggy, brittle) code and ensures that every (hard, pricey) act of improvement has a large and wide-reaching effect.

We’re not alone

It is not only creating software that needs be treated as an investment. In a true open source environment using others’ software, too, is an investment that we should make the most of. When we choose to depend upon existing free technology—in any way: WordPress, Xinha, Zope, Pylons, Trac—we will benefit ourselves and others by doing all that we can to improve it, participate in its communities of users and developers, and encourage the growth of those communities. The core reasons are the same as with our own code: the more eyes and hands that are on a piece of software, the better it can become.

There is, of course, an additional benefit to encouraging outside developers, namely lessening our internal development and maintenance burden by gaining volunteer contributors. When we create our own software we thus have the additional responsibility of building these communities from nothing. But when we are dealing with existing open source software this goal is all the easier to achieve. A community of developers already exists; our only responsibility is to gain admittance. We should treat our lack of commit rights not as an obstacle, but as a challenge.

Alongside the fuzzy feelings that come with earning the respect of an established group, there is, in keeping with the best of the open source mentality, great potential for everybody to win when we are welcomed into an existing community. We stand to reap the benefits of the efforts of the intelligent, experienced and caring people who develop the software that we have decided is worthy of our use, and we gain the potential to influence the path and priorities of their development with our input and interest. They, meanwhile, gain the feedback and work of a new set of smart (and funded) contributors with a real interest in the success of their project.

Finding those unicorns

I’ve framed this primarily as an economic argument to be concrete, but of course there’s more to it than that. The same stance is also about committing to respectful and positive relationships with outside agents; putting our faith in open and collaborative development environments and in the potential for groups of smart, committed people to produce great things even with differing agendas; taking the time and care to make our work as good as it can be and letting the rest of the world be the judge of it. It’s about being responsible consumers who contribute back to the products we consume, and being responsible producers who proudly stand by what we produce. And then there’s our aspirations at TOPP to do good and produce real positive change: all the time and money we can save by reusing and building on existing solutions can be diverted toward new projects to do even more good in the world, with cascading effects.

So let’s build and get involved with communities, submit and fix bugs in others’ code, give thanks and contribute ideas to the people whose software we use.

Let’s release Flunc, Externalator and Deliverance to the world for real, with welcoming websites and helpful documentation, and encourage people to use them and talk to us about them on mailing lists and IRC channels. Along with everything else it’s a good way to put our software to the test: to put a different spin on the dogfood concept, if no one else wants to use our software, why should we? Let’s keep up with Xinha development and submit behavioral patches to its authors instead of keeping our work isolated on our “vendor branch.” Let’s tell people about the recommendations of our smart interaction designers and submit our implementations of that advice; how many open source software communities have an excess of input from web software designers? Let’s release WordPress and Trac plugins to their wider communities and let’s solicit contributions and feedback to Listen and Opencore.

In short, let’s try to practice better open source. After all, we can only bring the open source ethos and philosophy to the broader world after we get it right in our software.

I thought it would be useful to make note of how a couple things were done recently using Deliverance.

The first is making http://nycstreets.org/help/ point to the main openplans.org help page. This was done ad-hoc, though I’ll show how to do the equivalent in a better way with the new openplans_hooks and build.ini.

The nycsr Zope lives at localhost:12002, while openplans Zope lives at localhost:13002. When a request comes into nycsr it is generally dispatched/forwarded to http://localhost:12002/VirtualHostBase/http/nycstreets.org:80/openplans/VirtualHostRoot/PATH. That path lets Zope know how the request was originally made (so that links don’t point back to localhost:12002).

You can do ad hoc overrides with dvhoster using remote_uri.txt. Specifically we want to forward http://nycstreets.org/help/PATH to http://localhost:13001/VirtualHostBase/http/www.nycsr.com:80/openplans/VirtualHostRoot/help/PATH — note that /help goes after VirtualHostRoot. To do this we change remote_uri.txt to look like:

  [{"path": "/help",
   "remote_uri": "http://localhost:13001/VirtualHostBase/http/www.nycstreets.org:80/openplans/VirtualHostRoot/help"},
   {"path": "", "comments": "main-site"}]
  

Note that paths have to be ordered with longest path first, as the paths are matched in order (and “” matches anything). The “remote_uri” key tells dvhoster where to dispatch the request. The value ultimately goes through openplans_hooks.py which gets another try at rewriting the paths; as implemented it only does this when remote_uri isn’t set.

If you wanted to mount a project this way, you could do it like this:

  [{"path": "/bikesforbob",
     "remote_uri": "http://localhost:14001/VirtualHostBase/http/www.nycstreets.org:80/openplans/projects/VirtualHostRoot/bikesforbob"}]

Another things we did recently is to put static files up at http://dev.nycstreets.org/nymap/. The files were checked out to nymap/src/nymap, and so the remote_uri is file:///path/to/base_path/nymap/src/nymap/. In this case we used build.ini, like:

   nymap uri = file:///usr/local/topp/dev.nycstreets.org/current/nymap/src/nymap/
   nymap path = /maps
   nymap project_local = false

This is similar to the ad hoc technique:

   [{"path": "/maps",
     "remote_uri": "file:///usr/local/topp/dev.nycstreets.org/current/nymap/src/nymap"}]

“nymap project_local = false” means that this is associated with the root of the site. If it was true then /projects/PROJECTNAME/maps would be mapped instead of /maps (in a virtualhost situation /maps would be matched regardless).

This got the maps in place, but they needed access to the artois to do XMLHttpRequests, and those can’t be done across domains. So we needed to proxy artois over to dev.nycstreets.org. This was done with:

   artois_proxy uri = http://artois.openplans.org/geoserver/ows/
   artois_proxy theme = false
   artois_proxy path = /maps/rpc
   artois_proxy project_local = false

Now requests to http://nycstreets.org/maps/rpc get proxied to http://artois.openplans.org/geoserver/ows. We don’t need to theme it because it’s all API calls. Turning off theming in the configuration is slightly more efficient (though only slightly as the XMLHttpRequest calls wouldn’t ultimately have been themed anyway).

Anyway, that’s a couple techniques used recently. This sort of thing can be used to do other manipulations with the site. I’m hoping this description will get people to consider this technique when other kinds of requests come in.

Okay, so maybe that’s a bit of a bold title, considering a) most of you have been using fassembler for a little while already and b) Ian wrote fassembler, not me.

However, it does feel like just today we’ve put the last little bit of polish on it, so that it’s officially ready for prime time for all of our deployment needs. There are still some parts of the process that can be improved, but it’s definitely workable, and it’s orders of magnitude easier to use than anything we’ve had up until now.

How, exactly, should it be used? Well, if you just want to use it to build a local development rig of the OpenPlans software, you can refer to the fassembler how-to. If you want to actually deploy a persistent site, one that lives on either our dev or our live server, you can refer to the OpenPlans deployment instructions.  The deployment instructions go into quite some detail about how our directories are laid out, and how you should manage the configuration and requirements for separate sites, and separate builds of the same site. Please do not try to do any deployment of any site on either flow or theman without reading this first.

Good luck and happy building!

Here are some preliminary results from a look at the openplans.org data. I should say that the values are approximate, but I think they should be pretty close to the real values.

Here is what it all means:

mem active - the number of active members that logged in to the site during the previous 30 days

mem avg life - the average number of days a member is active (estimated from all members who are now dormant)

proj active - the number of active projects who have had their wiki edited during the previous 30 days

proj avg life - the average number of days a project is active (estimated from all projects who are now dormant)

ml active - the number of mailing lists which received a post to the archives during the previous 30 days

ml avg life - the average number of days a mailing list is active (estimated from all those that are now dormant)

One thing to note is the sharp drop in active members in November. This is perhaps, in part, due to a migration of members from openplans.org to nycstreets.org. Also, of all the data points, the December value is the one I trust the least due to the way things are approximated. In January I plan on doing a bit of work to verify the accuracy of these estimates.

The mailing list activity is higher than I expected. I checked and made sure that this isn’t due to spam (spam doesn’t actually make it into the archive.)

So this is just the start of what we can do to analyze our usage information to help us make good decisions about the evolution of our software. From here we can go on to analyze Task Tracker and WordPress usage.

In his blog post, cabraham takes a stab at how we define success by saying:

Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.

My concern here is similar to my previously stated concerns that we are pulling each other in different and incompatible directions. TOPP is a large organization with a variety of projects under its wing, making it difficult to steer projects such that they complement each other rather than conflict. Using the same metrics for success for both NYCstreets.org and OpenPlans.org is an example of this.

A brief history…

While there has never been a clear definition as to what OpenPlans is or who it targets, one shared vision of OpenPlans was as a website geared at empowering citizens to organize and effect change, specifically in regards to issues of urban and community planning. This vision dovetailed nicely with the goals of our advocacy branch—as Streetsblog and Streetfilms developed the propaganda, OpenPlans.org would provide the tools. Yet OpenPlans.org was conceived of much too broadly to ever get very far in developing a community to complement those on Streetsblog and Streetfilms. As a result, NYCstreets.org was launched to combine the tools provided by OpenPlans with the focus of purpose that drive Streetsblog and Streetsfilms.

OpenPlans versus Openplans.org

As a consequence of the launch of NYCstreets.org, however, OpenPlans.org was stripped of a large part of its stated audience: no longer can OpenPlans.org build a community around urban planning issues without directly competing against its sibling, NYCstreets.org. Direct competition between the two sites hurts both and benefits neither.

There has always been confusion around the distinction between OpenPlans, the software, and OpenPlans.org, the website, and many have viewed them as two sides of the same coin. Although narrowly conceived, there was a never a problem with this view and we muddled along just fine. But now that we are running a service—NYCstreets.org—using the OpenPlans software stack we have an opportunity to reposition ourselves along the lines of the GeoServer model: we must reconsider OpenPlans as a product, capable of being easily installed and customized, rather than a tailor-made package designed solely to run our OpenPlans.org service and no more. The definition of success cabraham proposes is reasonable for measuring the success of OpenPlans.org, but to continue thinking of OpenPlans as having a one-to-one relationship with OpenPlans.org is short-sighted and will likely result in failure.

­The Success of OpenPlans ­

The possibilities of success are much richer for OpenPlans if conceived of as a product. We are no longer limited to the constraints of a single website with a single goal but instead have the opportunity to build software that can address many problems—ranging from facilitating Plone sprints as OpenPlans.org currently does, to organizing communities in the form of NYCstreets.org, or even facilitating the project management needs of small local governments. OpenPlans as a product can’t be over-designed and over-engineered to meet overly specific use cases, but should instead be built to be flexible, easy to use, and broadly useful.

Despite having an opportunity to broaden our scope, we must be careful to focus our efforts on improving the usability of our product. NYCstreets.org provides us the opportunity to explore real-world users and situations around which to build our software, and the existing OpenPlans.org website will continue to provide an entirely different set of examples. My suspicion is that we will learn that our model of segregating functionality by type (separating pages from tasks, for example) is not as useful as the BackPack model of allowing multiple content types on each page—after all, users don’t care about what type of data something is as much as the context to which it belongs.

It is my hope that OpenPlans will mature to the point that it can consider sustainability models like those proposed by Cholmes for GeoServer, perhaps even becoming as ubiquitous for project management as WordPress is for blogging.

Nick brought up some very good questions in his “Year in Review” post—none of which I intend to answer here.

Instead, I want to address a larger question he implicitly poses: how do we define success? At the moment we simply ignore the question altogether and are content to ride the wave of our day-to-day inertia, but it is a question that is crucial to our existence as an organization. No doubt we each have our own conception of what would make us successful: for some it is the quality of our software at the code level, for others the value it brings to our users, and for others still a sense of how ‘cool’ it makes them feel. Yet, without a shared vision of how we define success and the path to get there we will continue to be pull each other in different, often incompatible, directions.

Conventional measures of success like market share and profitability, however convenient and problematic they may be, simply don’t apply to a non-profit like ours. This puts us in the difficult position of having to define our own success. As Cholmes is so fond of saying, we’re not a non-profit but a ‘for-benefit’, so do we define success as the amount of ‘benefit’ we produce? How would you go about defining our success?

Coming up:

• Towards a Definition of Success
• Defining Failure
  

­

So, as 2007 draws to a close (I know it’s only November, but I’ve already bought all my christmas presents so I’m feeling a little ahead of schedule), it seems appropriate that we take a moment to look back on what we’ve accomplished this year, and think about where we’re headed for 2008.

Rather than try to sum up a year in my own words, I thought I’d put it out to you. Here are a few questions to get you started (relating to OpenCore/OpenPlans and TOPP):

• What are you most proud of from 2007?
• If you could change one thing about 2007, what would it be?
• What would you like to accomplish in 2008?
• Any new years resolutions?

The end result

Authentication and member management are among the first things that we want to pull out of our Plone-based stack and into their own application, deployed as either a microapp or a piece of WSGI middleware. A conversation I had with Nick Grossman on Friday helped me clarify my thoughts on how we should move forward with this, which I’ll try to summarize here. I’ll start by describing the end result I’m working towards, and then talk about steps to get there.

As an end result, I see us having two new pieces in our stack. The first, which I’ve been calling “Auther” in my head, handles the front end authentication handshaking. I had originally thought that we might use authkit for this (since it has a number of different auth types supported already), but David Turner tells me authkit isn’t very well regarded among the Pylons folks, and we’d be better off whipping something up for ourselves. In any event, there doesn’t have to be much here, this piece will just handle the basics of verifying credentials and generating cookies. When used in a WSGI context, it will intercept unauthenticated responses from other servers to provide challenges, and it will insert REMOTE_USER into the WSGI environment for those servers to honor upon successful authentication. The auth tool should not be tightly coupled to any particular type of storage, because the actual location of the user data will probably change as we progress.

The other new piece I’d like to see is what I’ve been calling “TeamRoller”. TeamRoller would basically be a Pylons implementation of the TeamSpace functionality. It would handle all of the team and membership management, including what roles each user has within a team. It would also understand sites, so that members or teams can be said to belong only to a specific site, or can be shared across sites.

Like Auther, TeamRoller would work either as a microapp or as a piece of WSGI middleware. When used as a microapp, there would be a REST API to access and manage people, teams, and the relationships between them (memberships, roles, etc.). There would also be an HTML interface, modeled after the interface currently in use on openplans.org. When used as a WSGI app, it would populate the WSGI environment with information about the REMOTE_USER’s team roles and affiliations for use by downstream applications. Downstream apps would also be able to import a TeamRoller library to retrieve other team and team membership related information.

Currently all of this is happening inside of Zope; there are PAS plug-ins that handle cookie generation, authentication, and the injection of local roles for a given user based on her team membership. The member, team, and team membership data is all stored as content in the site; the plug-ins interact with this content to get their information. What we want to do is get from here to there quickly but smoothly, and with minimal risk.

One step at a time

I propose we do so by making a lot of small changes, deploying something new every week or two, spreading the transition out over the course of a couple-few months. I don’t have things thought through all the way to the end, but I do have some ideas on how to start.

First, we get a basic Auther implementation working. All of our applications already honor the same cookie, all we have to do is get Auther to generate that cookie and the other apps will work just as before. Things will need to be set up so that the login form comes from Auther; either the apps will explicitly specify an Auther URL as the login page, or possibly Deliverance’s VHoster would handle that. At this point, all member data and team data is still inside Zope; Auther will actually call our remote auth API to verify the credentials.

The next step would be to start moving the basic user authentication data out of Zope. Auther would grow its own very simple database, just storing usernames, password hashes, a UUID for each user, and a list of the sites with which the user is associated. Ultimately this data will probably be coming from TeamRoller, and the database should be put together with that in mind, but we can just stuff it into Auther as a starting point.

At this point, there would still be an OpenMember object in Zope, but it would no longer be providing authentication. Both openplans.org and nycstreets.org could be driven by the same Auther rig, so we could get rid of the remote auth hack that I just finished putting together. The member objects in the ZODB would still be providing most of the member data, which would be queried via the catalogs, as before.

The next incremental step, then, would be to work on pulling this data out, so that our member data comes from the same database as the authentication info. By this time, the TeamRoller data model would need to be fairly refined, so we can start populating it with data. We may even be deploying TeamRoller, although it wouldn’t yet be doing everything that it will ultimately do. The current PAS plug-ins that we’re using, which pull member data from the OpenMember objects, would be replaced with plug-ins that get that data from TeamRoller. All of the Zope code that retrieves info from (or writes info to) the member object would be changed to work with the user object. We could actually start on this even sooner, since the current PAS plug-ins support this as well.

The step after this would be to start using TeamRoller to manage the teams and team memberships. TeamSpace at this point would be replaced by TeamRoller, and much of what used to be in Zope would now be outside of it. The borg.localrole PAS plug-in that is currently pulling security info from the TeamSpace infrastructure would go away, to be replaced by a plug-in that used TeamRoller.

Subject to revision

This is obviously not fully baked; I very much expect that things will change as we move forward. The greater the number of steps in the future, the greater the likelihood that things will diverge from the plan between now and then. This is okay, though; that’s how agile programming is supposed to work, and it’s the very reason why we take lots of little steps instead of fewer large ones. Other things to throw into the mix as we go is moving from Plone 2.5 to Plone 3, so we can start using Paste as our http server for Zope, can start using the handy GS upgrade infrastructure, have the option of using Zope in a WSGI context, etc. Not sure how or where this fits in. But I think we have a little bit of time to get to this point, since the first two steps are right in front of us, clearly defined, and they have to be completed and deployed before we start working on the next stuff.

i18n stuff

Most of the opencore package has been marked appropriately for i18n translation. A fake “az” translation has been created to serve as a test language for flunc. Flunc has been modified to test off this translation by default. A utility has been created to assist in keeping the “az” language up-to-date. For a full explanation see here.

Right now I’d like to branch the ftests to replace all the hardcoded text with the appropriate message ids. From here we could extend the “az” translation to sub in message variables values so that flunc can test against them as well.

opencore.stats

The groundwork is in place with this to support drilling into the ZODB to reveal useful statistics on opencore site usage.

TOPP News #2

On it’s way… out… the … door.