OpenCore Software

Robert and I have been working on the opencore feeds. Currently, these are only used by the project summary page, but looking at the site, we have many things that could be feeds: Latest Projects, Latest Members, Recently Updated Projects, just to name a few. This blog post is sponsored by the word ‘agnostic’, where agnostic means “optimistically not incestuous code” and “incestuous” means “code that unfairly presumes coupling or functionality that destroys perceived modularity and forces binding to a framework even when the desired functionality should not depend on the framework”. Okay, those are just words. The idea is that incestuous code is badly coupled code. Its a word I use alot to describe code I don’t like, because something can (perhaps wrongly) be described as modular but be completely incestuous. Agnostic is the antidote (and the antonym) to incestutous.

Sorry for that gratuitous introduction. The point is that I think what we did with feeds is not a bad model:

[So I tried to upload this image through wordpress’s xinha.  Of course, horrible things happened, so now I’m just linking to it.  GUIs are my bane; did i mention I’m now editting this in html mode, as I almost always do, even though basically i want text ]

This is a horrible diagram with symbolism that only makes sense to me. So let me describe. You have a bunch of content (projects, mailing lists, etc) that provide feed data. They may provide feed data in more than one way, but that’s another story. Instead of having the wild west, I make an interface called IFeedData that forces the feed in the standard format. In this way, I have decoupled what the feed comes from from what a feed is. IFeedData (and its items) have some things that are mandatory to define — the essence of what a feed is — and what is optional.

Moving further right on the chart, there are a bunch of templates. Any of these templates can be used to render any feed. Some of them will work better with particular type of feed than others — for instance, if the feed doesn’t have icons, the portrait_feed_snippet.pt is pretty silly; but each can render the basic functionality of “what a feed is”, and may support optional items, conditionally displayed if they exist. To extend functionality, add a new template.

So we have feed providers, which give us feeds, and templates, which display feeds. How are they used?

So we have this viewlet manager, which has a bunch of viewlets. The viewlets have some sort of context which gets adapted to give a feed. The viewlets *also* have something that says “use *this* template”. In other words, the viewlets decide how the feed is displayed.

I’m not writing this to say “ooooh! look at this cool pattern i helped make!” I’m writing this as an illustration of how something that isn’t (amazingly) incestuous can work. Notice you have decoupling on every level. You have content providers which have rules to transform them to feeds. You have feeds, which have various templates that can be used to display them. You have viewlets, which contain the display logic, tell how to adapt their context to the feed, and choose the template. On top of that, its a simple system: abstract enough to handle anything I can imagine atm, but concrete enough to actually use.
There are problems going forward that I haven’t conquered yet, but I am at least comfortable with the pattern.

We’re at PyCon! The lion’s share of the openplans / opencore developers are here:

Ian Bicking, Josh Bronson,  Mel Chua, Jeff Hammel, Anil Makhijani, Robert Marianski, Doug Mayle, Whit Morris, and me (Paul Winkler). Look for us and say hi!  We are friendly and mostly harmless. Oh, and we might have a job or two open.

I’m giving a talk Friday at 3:55, and Ian’s doing one on Sunday at 11 AM

Seems to be a theme in my reading this week: Dumb ideas are often the best ideas.

Friday, Rolando posted a link to Paul Graham’s recent “Six Principles of New Things”.

“… this is practically a recipe for generating a contemptuous initial reaction. Though simple solutions are better, they don’t seem as impressive as complex ones. Overlooked problems are by definition problems that most people think don’t matter. Delivering solutions in an informal way means that instead of judging something by the way it’s presented, people have to actually understand it, which is more work. And starting with a crude version 1 means your initial effort is always small and incomplete.”

Joel Spolsky probably read it too, because he just posted an article about some notable successes that he initially greeted with “That will never work”.

I’m also reminded of Farnsworth’s breakthrough with the invention of television. Several competing labs were trying to invent a working television; Farnsworth had a small shop with minimal funding. One of the critical hurdles was a seemingly impossible bit of glasswork for the camera tube. Farnsworth’s self-taught brother-in-law eventually succeeded at this task in part because he didn’t know any better and tried something that shouldn’t have worked. (At least that’s how I remember the version of the story told in Sorkin’s The Farnsworth Invention.)

All of which begs the semi-kidding question: are we doing something impossible enough?

To make for a smoother deployment next time, I’ve thought of a few things:

• we need to write more tests and we need to run them; when I joined the firefighting of the 0.9.7.7 deployment, the unit and flunc tests were broken in several spots and I suspect they were broken when the site was deployed; also, there were some things like the password request form that had a runtime error and would have been easily caught had there been a test for it
• we need to remember to merge fixes between svn release branches and the trunk; there was at least one bug that was fixed in the trunk but not merged into the release branch and I believe there were some things that were fixed in the branch but not merged to the trunk; this can get complicated because when you fix a bug you need to fix it on both the branch and trunk and be able to run the tests on both to make sure you haven’t broken anything
• when using trac, I found it useful to specify what site the bug was found in and what release of opencore was currently running on the site; after fixing it, I would record both the trunk and branch rev number that I made and then post another entry when it was deployed and verified on the live site and only then would I close it; this last step might only be appropriate during “firefighting” mode
• finally, I think when we are firefighting, having frequent standing meetings, as we did do, is helpful

A lot of times at the Snow Sprint I was struck by a feeling that I was in the middle of a group of people who were really doing open source right. So on the plane home I started thinking about what that meant and why I haven’t felt that way at TOPP.

Well, a lot of us have said at various times that our attempts at doing open source development have often been frustrating and rarely seem to be worth the effort. Coming out of the sprint, I think that’s because we are missing a very big piece of the “True Spirit of Open Source”. Doing open source properly is not just a matter of putting the GPL on our code—it is not about making a legal contract but about making a social one. It is about committing to reuse and collaboration both internally and externally and it is well worth the effort.

Writing, finding, and using software all take time and care and with someone paying for our time and care we ought to think of these acts as investments. We should make the most of those investments by seeking code reuse and encouraging collaboration. The more we encourage heavy use of software and heavy reuse of code, the better our ability to improve and solidify it. Whether or not we’ve read Dreaming in Code (though everyone here should—as soon as possible—seriously) we all know that software is hard to do well. It’s full of unexpected behavior, prone to breaking when you touch it, and rarely optimized. We can improve it only by testing and exercising it: give it to people to use, poke it in new ways, put it under stress, see what breaks, fix it and don’t let it break in the same way again.

Laziness is a virtue

Because software is expensive to write, it is most responsible of us to write as little as we can. And because software is hard to write, it’s rarely done once it’s written; most software has tremendous room, and often necessity, for improvement. So our goal should be to maximize the impact of every improvement we make. This primarily means seeking to increase the breadth of our code’s effect: convincing other people to use our software, sharing code across multiple products, and reusing products in as many ways as we can across everything we develop.

To give a specific, if vacuous, example, we have five or more different pieces of code deployed on OpenPlans that generate and send email to users: Listen sends mail, Twirlip sends mail, WordPress sends mail, and Opencore sends mail in at least two different ways. Suppose, instead, that we had a single shared way to send mail to users. Aside from the immediate benefit of less overall knowledge to store, the different mail-sending needs of all these applications would force us to develop a stable and reliable method for sending mail, and when heavy usage of one of those dependent applications inevitably reveals an inefficiency or bug in the mail-sending code, multiple otherwise unconnected parts of our site would instantly benefit from our resulting improvements. Put that same mail-sending code on two or more websites and encourage other developers to deploy it in their applications, and we can tremendously increase both the number and variety of stress points and the magnitude of impact of each strengthening, both the opportunity and the payoff for improvements.

So maximizing adoption of every piece of software by both users and developers is a forward-thinking and cost-effective stance. It gives programmers the most opportunities to strengthen their (buggy, brittle) code and ensures that every (hard, pricey) act of improvement has a large and wide-reaching effect.

We’re not alone

It is not only creating software that needs be treated as an investment. In a true open source environment using others’ software, too, is an investment that we should make the most of. When we choose to depend upon existing free technology—in any way: WordPress, Xinha, Zope, Pylons, Trac—we will benefit ourselves and others by doing all that we can to improve it, participate in its communities of users and developers, and encourage the growth of those communities. The core reasons are the same as with our own code: the more eyes and hands that are on a piece of software, the better it can become.

There is, of course, an additional benefit to encouraging outside developers, namely lessening our internal development and maintenance burden by gaining volunteer contributors. When we create our own software we thus have the additional responsibility of building these communities from nothing. But when we are dealing with existing open source software this goal is all the easier to achieve. A community of developers already exists; our only responsibility is to gain admittance. We should treat our lack of commit rights not as an obstacle, but as a challenge.

Alongside the fuzzy feelings that come with earning the respect of an established group, there is, in keeping with the best of the open source mentality, great potential for everybody to win when we are welcomed into an existing community. We stand to reap the benefits of the efforts of the intelligent, experienced and caring people who develop the software that we have decided is worthy of our use, and we gain the potential to influence the path and priorities of their development with our input and interest. They, meanwhile, gain the feedback and work of a new set of smart (and funded) contributors with a real interest in the success of their project.

Finding those unicorns

I’ve framed this primarily as an economic argument to be concrete, but of course there’s more to it than that. The same stance is also about committing to respectful and positive relationships with outside agents; putting our faith in open and collaborative development environments and in the potential for groups of smart, committed people to produce great things even with differing agendas; taking the time and care to make our work as good as it can be and letting the rest of the world be the judge of it. It’s about being responsible consumers who contribute back to the products we consume, and being responsible producers who proudly stand by what we produce. And then there’s our aspirations at TOPP to do good and produce real positive change: all the time and money we can save by reusing and building on existing solutions can be diverted toward new projects to do even more good in the world, with cascading effects.

So let’s build and get involved with communities, submit and fix bugs in others’ code, give thanks and contribute ideas to the people whose software we use.

Let’s release Flunc, Externalator and Deliverance to the world for real, with welcoming websites and helpful documentation, and encourage people to use them and talk to us about them on mailing lists and IRC channels. Along with everything else it’s a good way to put our software to the test: to put a different spin on the dogfood concept, if no one else wants to use our software, why should we? Let’s keep up with Xinha development and submit behavioral patches to its authors instead of keeping our work isolated on our “vendor branch.” Let’s tell people about the recommendations of our smart interaction designers and submit our implementations of that advice; how many open source software communities have an excess of input from web software designers? Let’s release WordPress and Trac plugins to their wider communities and let’s solicit contributions and feedback to Listen and Opencore.

In short, let’s try to practice better open source. After all, we can only bring the open source ethos and philosophy to the broader world after we get it right in our software.

Robert pointed out that our “microapp” architecture has the potential to become extremely hard to work with and isn’t really giving us any benefits in the process, since we’re not actually decoupling any of the components. I’d like to think more about that, but in the meantime, I’ve been thinking about what I think would be a really compelling use for proper microapps on our site that wouldn’t be increasing our administrative burden in the way Robert talks about.

(Incidentally, just to be clear, when I talk about proper microapps, I mean simple UI-less applications with a very stable API over HTTP that provide a single service intended to be plugged in to other contexts.)

So, we’re trying to use (quasi-)microapps to build our site on the backend, but I think microapps would be most valuable as services we provide *for users* instead. It seems more compatible with the microapp tendency towards runtime configuration over HTTP; while we end up bending the code and our builds to configure these applications to set up our site at “build and compile” time, users could just configure the applications for themselves on an already-running site to gain additional and personalized functionality.

Offered as user facing tools, these microapps would provide extremely flexible functionality beyond what we offer universally — ways for individual users to layer their own custom configurations on top of the site. Transcluder is a simple example; by just turning on a sitewide transcluder, we would allow users to build very flexible componentized webpages. I think an HTTP request hub like Cabochon could be another really powerful service for users: while we handle our own events internally with code and configuration, users could configure a set of individually saved hookups between HTTP events and resources. So one user could set up a rule: “When I publish to the project’s blog, mark my `write blog post` task as completed” while someone else sets up a rule that “When anyone edits this wiki page, create an account page message for me.” Throw in a cron-like microapp to schedule HTTP requests and now users can have a new “Write weekly blog post” task created every week and more.

Admittedly I’m doing a tremendous amount of hand-waving here (authorization? authentication? request interception? recursive requests? — implementation details!) and I haven’t come up with very compelling use cases, but I don’t think it’s impossible and I think the real potential of the idea is that *we don’t have to come up with the use cases* — people could come up with really interesting combinations of events far beyond anything we’d want or need to build into our software.

Another big advantage here is that since these microapps would have very well-defined APIs that work over HTTP, we could actually deploy them with minimal user interfaces and gradually build up UI over time. “Power users” would immediately get to experiment with the tools immediately, and we could even use their feedback to determine common usage patterns to inform the friendlier interfaces. So we can turn on Transcluder without any UI, and people could immediately start using it by editing a bit of HTML; then, as a first pass at UI, we could build a switch into the Xinha linking interface to toggle the “rel=’include’” attribute; and eventually we could design a much more sophisticated UI (perhaps leaving the “design-less” interface available underneath in the same way that WYSIWYG editors allow you to drop down to HTML) but without that design ever being a bottleneck to deployment or usage by at least some people. Likewise an event hub could start out by letting people input URLs and HTTP verbs, and eventually build a descriptive interface which lets people connect “things to do” from a list.

I think this would be a really powerful way to add, essentially, arbitrary flexibility to a site without having to hard-code it, and without having to come up with individual use cases, instead offering the tools to users and letting them come up with their own uses. And since we really could treat it as a layer on top of our software, which we don’t actually have to know anything about the use of, it wouldn’t be adding to our administrative burden, since we wouldn’t have to worry about the connections between microapps. I don’t think it replaces the need for some sort of internal connection between features and data — we can inevitably come up with much more tightly integrated services on the backend, and in any event I’m sure there are commonly useful hookups which we ought to just provide directly. But it would be a great way to provide functionality beyond what we decide to offer or what occurs to us to implement.

Here are some preliminary results from a look at the openplans.org data. I should say that the values are approximate, but I think they should be pretty close to the real values.

Here is what it all means:

mem active - the number of active members that logged in to the site during the previous 30 days

mem avg life - the average number of days a member is active (estimated from all members who are now dormant)

proj active - the number of active projects who have had their wiki edited during the previous 30 days

proj avg life - the average number of days a project is active (estimated from all projects who are now dormant)

ml active - the number of mailing lists which received a post to the archives during the previous 30 days

ml avg life - the average number of days a mailing list is active (estimated from all those that are now dormant)

One thing to note is the sharp drop in active members in November. This is perhaps, in part, due to a migration of members from openplans.org to nycstreets.org. Also, of all the data points, the December value is the one I trust the least due to the way things are approximated. In January I plan on doing a bit of work to verify the accuracy of these estimates.

The mailing list activity is higher than I expected. I checked and made sure that this isn’t due to spam (spam doesn’t actually make it into the archive.)

So this is just the start of what we can do to analyze our usage information to help us make good decisions about the evolution of our software. From here we can go on to analyze Task Tracker and WordPress usage.

What’s our definition of success, our institutional purpose? Personally I have a very clear sign of success in my own mind: how strongly do I recommend our service to other people? I’m a a bit shy about suggesting people use “my” stuff, where “my” means something for which I feel responsible for both in its success and failure (which of course includes openplans.org). If I personally felt confident recommending openplans.org to anyone, especially people who I don’t want to give personal technical support, then I would feel part of a successful project. I’d love it if then a lot of people everywhere used it, but my personal feeling of satisfaction is not contingent on that. Or maybe that’s what my notion of success+1 would become.

What about the development analog: would I recommend our software to other developers (not hosted by us)? Frankly I don’t care much about this. Making our software generally usable I think is an important discipline for us, but to me it’s not an end. Why not? I’m not really sure.

Sobering thought for the day… there are two kinds of people: the disabled, and the temporarily abled.

With that in mind: “Degrading gracefully” for non-javascript browsers isn’t just a theoretical nicety. For some people it’s the only way they can use a website at all.

I’m a javascript novice, so I hardly know a lot about the best practices here, but there’s some interesting commentary out there if you google a bit. For example, “Test your pages, you wanker. With assistive technologies. And real disabled people. Or you’re a bigger nob than you look.”

Neither am I an expert on accessibility, disability rights, etc. (but my mother-in-law is).

In a funny way, we’re lucky that our current functional testing technology doesn’t support javascript, because it ensures that things mostly work.  But for a concrete example - the one that prompted this post - can flunc “see” a textarea if you style it as hidden by default? I suspect it doesn’t care about css-driven visibility; all it does is find the element in the document.

In his blog post, cabraham takes a stab at how we define success by saying:

Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.

My concern here is similar to my previously stated concerns that we are pulling each other in different and incompatible directions. TOPP is a large organization with a variety of projects under its wing, making it difficult to steer projects such that they complement each other rather than conflict. Using the same metrics for success for both NYCstreets.org and OpenPlans.org is an example of this.

A brief history…

While there has never been a clear definition as to what OpenPlans is or who it targets, one shared vision of OpenPlans was as a website geared at empowering citizens to organize and effect change, specifically in regards to issues of urban and community planning. This vision dovetailed nicely with the goals of our advocacy branch—as Streetsblog and Streetfilms developed the propaganda, OpenPlans.org would provide the tools. Yet OpenPlans.org was conceived of much too broadly to ever get very far in developing a community to complement those on Streetsblog and Streetfilms. As a result, NYCstreets.org was launched to combine the tools provided by OpenPlans with the focus of purpose that drive Streetsblog and Streetsfilms.

OpenPlans versus Openplans.org

As a consequence of the launch of NYCstreets.org, however, OpenPlans.org was stripped of a large part of its stated audience: no longer can OpenPlans.org build a community around urban planning issues without directly competing against its sibling, NYCstreets.org. Direct competition between the two sites hurts both and benefits neither.

There has always been confusion around the distinction between OpenPlans, the software, and OpenPlans.org, the website, and many have viewed them as two sides of the same coin. Although narrowly conceived, there was a never a problem with this view and we muddled along just fine. But now that we are running a service—NYCstreets.org—using the OpenPlans software stack we have an opportunity to reposition ourselves along the lines of the GeoServer model: we must reconsider OpenPlans as a product, capable of being easily installed and customized, rather than a tailor-made package designed solely to run our OpenPlans.org service and no more. The definition of success cabraham proposes is reasonable for measuring the success of OpenPlans.org, but to continue thinking of OpenPlans as having a one-to-one relationship with OpenPlans.org is short-sighted and will likely result in failure.

­The Success of OpenPlans ­

The possibilities of success are much richer for OpenPlans if conceived of as a product. We are no longer limited to the constraints of a single website with a single goal but instead have the opportunity to build software that can address many problems—ranging from facilitating Plone sprints as OpenPlans.org currently does, to organizing communities in the form of NYCstreets.org, or even facilitating the project management needs of small local governments. OpenPlans as a product can’t be over-designed and over-engineered to meet overly specific use cases, but should instead be built to be flexible, easy to use, and broadly useful.

Despite having an opportunity to broaden our scope, we must be careful to focus our efforts on improving the usability of our product. NYCstreets.org provides us the opportunity to explore real-world users and situations around which to build our software, and the existing OpenPlans.org website will continue to provide an entirely different set of examples. My suspicion is that we will learn that our model of segregating functionality by type (separating pages from tasks, for example) is not as useful as the BackPack model of allowing multiple content types on each page—after all, users don’t care about what type of data something is as much as the context to which it belongs.

It is my hope that OpenPlans will mature to the point that it can consider sustainability models like those proposed by Cholmes for GeoServer, perhaps even becoming as ubiquitous for project management as WordPress is for blogging.