Scope

I’m just talking about opencore, the Plone extension at the heart of the openplans software stack.

I’m not talking about any other part of the openplans software stack (tasktracker, wordpress, deliverance, …)

I’m definitely not talking about openplans.org the website.

Opencore needs a release manager?

Opencore was originally created by TOPP for www.openplans.org. So opencore releases have historically been driven by the needs of www.openplans.org’s deployment schedule. Since our deployment process doesn’t require a formal opencore release, releases tend to be a rare afterthought.

But there are at least two things that more formal releases would help with:

1. Lower the barrier for other community members to get started using opencore.
2. Let the world know about what we’re doing.
   

My overarching goal is to promote a lively community of developers and users for opencore, and provide some balance to TOPP’s path of least resistance–to only worry about our own deployments.

Why me?

Ethan asked me to do it and I was happy to volunteer. (If you don’t know who Ethan is either, this picture should suffice: http://www.openplans.org/people/ejucovy )

Who?

I’m Paul Winkler, a Zope user and developer since 1999, a TOPP employee since October 2007, and Opencore’s release manager since right now. http://www.openplans.org/people/slinkp

What will determine the release schedule?

I want to decouple Opencore’s release cycle from the TOPP deployment cycle. Douglas Mayle has volunteered to act as the openplans.org deployment manager; he and I have tentatively agreed on this plan:

• Opencore releases will be on a time-based schedule, TBD.
• Features that aren’t ready in time for a release can just wait until the next release.

  Rationale: Feature-driven releases inevitably get delayed and delayed, and the features don’t get done any faster.

• Openplans.org deployments will be on an independent time-based schedule. (I believe the plan is to deploy weekly.)
• “Must-have” openplans.org features and fixes can be deployed independently of the opencore release cycle by putting them in a new package that layers on top of opencore.
• Openplans-specific code and text (like the current copyright notice) will be gradually moved out into this new package.
• The aforementioned openplans.org customization package does not exist yet; we need to create one ASAP.
  

Of course TOPP’s own needs will continue to drive a lot of opencore development, which is a good thing; but that shouldn’t be a burden on you, the wider opencore user/developer community.

Does the Release Manager have to be a TOPP employee?

No. I’m just the first. It should be a rotating job. It’s my hope that before long, people outside TOPP will be willing and able to play this role.

When will the job rotate?

Don’t know yet.

What does the Opencore Release Manager do?

• Manage the release schedule (not TOPP’s deployment schedule!). This does not mean I’m the dictator, just the point man and coordinator.
• Shepherd release candidates through a stabilization process.
• Create and announce final release packages.
• Document upgrade process.
• Document any backward incompatibilities.
• Draft and enforce relevant dev process standards (e.g. which branches or plugins should you commit bugfixes or features to).
• Document this process for the next release manager.

At this stage, I’ll also be acting as a sort of ambassador between TOPP and the wider community. Which is a bit silly, as every opencore contributor does that to whatever extent they wish. But I think for now it’s useful to consciously formalize it as part of my job. I will make an effort to:

• Solicit community input about decisions we need to make for opencore.
• Remind TOPP employees to discuss and make decisions about opencore out in the open. If you have a real-life relevant conversation I’ll try to remind you to post a summary to the mailing list.
• Keep the community updated on changes to our build process. If we break your builds, you can complain to the opencore-user or opencore-dev mailing list, and you should expect a response from me.
• Lobbyist for the community: e.g. prioritizing and scheduling feature requests and bugfixes.

There will of course be other forces at TOPP driving things with other priorities, but *my* role will be to ensure that the community’s needs don’t get ignored. This is more about release management than development management: I’m not in charge of deciding which features get done when. But whenever good code comes from the community, we have a responsibility to get that code released in a timely manner, and generally be as responsive as we can.

What about the rest of the stack?

We realize that most of the non-TOPP users of opencore are using some or all of the entire www.openplans.org stack. We’d eventually like to be making formal releases of all the packages, and possibly some kind of “batteries included” big meta-package; but that’s in the future. For now, opencore is our fastest-moving target and the most in need of release management.

Release Numbers

We supposedly have a policy now (Ethan proposed it here: http://tinyurl.com/4ybpro) … but we haven’t been sticking with it. I’m going to draft a proposal for a simpler convention and process. More on this in a later message.

Releases, Eggs, and Builds

I’d like to be releasing versioned eggs of opencore to PyPi. These should be installable into a suitable Zope/Plone instance using nothing more than easy-install.

(We have made a few releases to PyPi, but they aren’t actually usable … some files are missing.)

For bootstrapping a full stack including Zope, Plone, and other non-egg stuff, TOPP will continue to use and develop our Fassembler build tools. But it should become easier to do without that, if you want.

Branch Policy

We are currently trying to follow a “Stable Trunk” practice. Experimental and risky code should not be on the trunk.

Developer Infrastructure

This is only tangentially related to release management. I just wanted to note in passing that we’re planning a more thought-out integrated home on the web for opencore development. We’re calling this the “TOPP Dev Center” for now. Other open-source packages from TOPP will live there too.

In the meantime:

Wiki and mailing lists are at: http://www.openplans.org/projects/opencore

Trac (bug tracking) is at: http://trac.openplans.org/openplans

If you need a Trac account, just ask. (I wish we could take anonymous bug reports, but we got trac spam.)

If you want SVN commit access, first you should send some bugfixes and features as patches (either in trac or on the dev list). We’ll happily give commit access to people who have a history of submitting good patches. There will be a contributor agreement you’ll have to sign, somewhat similar to Zope’s.

Current Road Map

Some things coming up in the short term:

• Plone 3

  We will soon be releasing a version of opencore that runs on Plone 3. Rob Miller has done all the hard work and we just need to QA this branch and merge it back to the trunk.

  Currently, the plan is for TOPP to create a release candidate from this branch, QA it, and deploy it to www.openplans.org. When that goes well, I will:
  

  1. Branch off the current opencore trunk to an opencore-plone2.5 branch
  2. Make a final stable release from the opencore-plone2.5 branch (to be followed by bugfix releases as necessary) 
  3. Merge the plone 3 branch back to the trunk. 

  This will definitely happen this summer; more concrete dates will be forthcoming. 

  If people in the community expect to continue using the Plone 2.5 version for a while, I would like to hear about it so I can serve you better.

• Easier “plugin” management using zc.autoinclude. This will replace our usage of zcmlloader, doing the same job but better (and other people are actively developing it).
  

Some ongoing maintenance tasks I’d love community help with:

• Opencore is slow. Let’s make things faster. (But let’s get plone 3 merged to trunk first, it already helps substantially.)
• The test suite has a lot of problems. More on this in a later message.
• Dependency cleanup. I suspect there is cruft in the Products “bundle” that we don’t actually use.
• What do you guys want?

Going forward, this road map should have a permanent home prominently linked in the aformentioned dev center. There is already such a page at http://www.openplans.org/projects/opencore/planning but it badly needs an update. I’ll be looking to revive that.

Questions?

Whew. Sorry for the length. Feedback would be most welcome.

- PW

At the end I almost talked myself into refactoring stuff to make the good parts of fassembler usable in zc.buildout, so that we could move over to that in the future if we want to.  Then… right at the end… pow!  Files!  All files, only files!

What if our build process just built files?  Relocatable files.  Not-system-dependent files.  You could tar the whole thing up, copy it to another location or server, unpack it, and get the exact same site.  That tarball would basically be equivalent to the build itself.

It’s not that App Engine seems particularly awesome.  More that it seems like it might be Just The Right Kind Of Dumb.  It’s that this makes it so easy to think about the build, about what it does, about stuff like the workflow for testing and deployment and rolling back.  Easy to explain to other people, easy to audit, easy to debug.

The problem of course is that it requires pushback into other products.  Lots of other products.  But we can do that.

Thoughts?

The presentation this week was by Peter Fein about GrassyKnoll, a text search engine written in Python.

From TOPP’s point of view there are several interesting things about it:

• It could provide us with a sitewide search solution for openplans.org and livablestreets.com.
  
• It can use any of a number of pluggable back ends (notably PyLucene).
  
• You interact with it entirely via a simple REST API.
  
• A GrassyKnoll client can also trivially be a GrassyKnoll server. (Peter gave a live demo of this.) In theory this could allow for fun things like smart clustering, where one server gets your query and dispatches queries to N other servers, and then merges the results appropriately. The end client still sees the same simple rest API.
• It’s written entirely in Python and is Free.
• They’re looking for more people to help out and would love for us to get involved.

The major down side is that they’re still some months away from a production-ready release. (There were a few glitches in the live demo.) When I pressed him about this, Peter said “definitely by the end of the year.” I got the impression that more hands helping would speed that up.

Also, it doesn’t define any kind of common query language; it just passes them along to the back end. So you do have to know what you’re really talking to.

Peter said he was hopefully coming to the next NYCPUG meeting that we’re hosting here at TOPP on July 15.

As a result, all of opencore’s page templates are parsed in ZPT’s “html mode”. This makes for some surprising validation behavior (e.g. unclosed tags is not an error, but putting content in a “link” tag is an error… which sucks if you happen to be trying to generate georss and not HTML …)

I ended up not even using ZPT for my georss feed, but for posterity, there are at least two possible workarounds:

1. Put an xml declaration at the top of your template (this makes the parser to use XML mode). If  your template is only for a snippet and not for a whole document, you don’t want the declaration showing up in the middle of the final document. To avoid that, use METAL macros.  It’s a bit tricky to do this from python, so we have a handy render_tal function in opencore.browser.tal.  And you’ll need to manually encode to your output encoding if you want to use this with response.write().  An example of this that I later reverted can be seen here.

2. You could, if you don’t want to use macros and can’t have the xml declaration in your output, instead subclass Five’s ZopeTwoPageTemplateFile like so to force the TAL parser to use the right mode. (Thi sis also in the example linked above.)

 class XmlPageTemplateFile(ZopeTwoPageTemplateFile):

    """This class exists because the process whereby PageTemplateFile
    decides whether to use HTML or XML mode for parsing and validation
    is completely stupid.
    """

    content_type = 'text/xml'

    def _cook_check(self):
        # In PageTemplateFile, this gets called potentially multiple
        # times (if you're in dev mode), sniffs the file's content,
        # and if it doesn't start with an XML declaration, uses HTML
        # mode.  We don't want that.  So here, we duplicate the same
        # code without the sniffing.
        if self._v_last_read and not DevelopmentMode:
            return
        __traceback_info__ = self.filename
        try:
            mtime = os.path.getmtime(self.filename)
        except OSError:
            mtime = 0
        if self._v_program is not None and mtime == self._v_last_read:
            return
        t = self.content_type  # Take my word for it, dammit!
        f = open(self.filename, "rb")
        if t != "text/xml":
            # For HTML, we really want the file read in text mode:
            f.close()
            f = open(self.filename, 'U')
            text = ''
        text = f.read()
        f.close()
        self.pt_edit(text, t)
        self._cook()
        if self._v_errors:
            logger.error('Error in template: %s' % 'n'.join(self._v_errors))
            return
        self._v_last_read = mtime

For my georss feed, I ended up deciding this was all too much bother and I just went back to using Python string interpolation. There just wasn’t enough to the “template” to justify the pain of getting TAL to work properly under these constraints.

It could easily be generalized, there’s nothing really zope-specific about it, all it really cares about is a path to a common-apache-format access log and the PID of a process to watch memory. The result is a nice little Flot graph where you can select a range and see what URLs were hit during that time.

Here’s a screenshot of an example from a flunc run against opencore (the “basic” suite):

So what does this tell us really? I’m not sure. Multiple runs show that the graph generally starts around 400 MB and ends around 500 MB, but what happens between does not seem consistent. So I’m not sure if this little app is really that useful for opencore.

There were a number of talks, panels, and conversations, but the central focus of the event was a contest between 21 featured projects. Early on the first day, each of these projects was given 2 minutes to describe themselves; this was immediately followed by an hour-long session where attendees could walk around and chat w/ project members to learn more. Near the end of the last day, they were all given another 2 minutes to make their closing pitch, after which attendees were able to vote for which projects they felt were most deserving. Each person was given 3 votes, which could be distributed as desired (i.e. all for one project or spread among 2 or 3). the top three vote winners received $25000, $15000, and $10000, respectively; the rest split equally the remaining available grant money, which rumor had it came to about $2000 per.

It was a lot fun and inspiring to see what good work people are doing in the world. The coolest thing I saw was by Ushahidi, the winner of the $25000 prize. They’ve been mapping outbreaks of violence and peace as a reporting and research tool during Kenya’s post-election tumult. They’ve got a great timeline map, with which you can get a great visual representation of the way events developed through time. The timeline component is something out of MIT’s SIMILE project, and thus is free for us to use. This could be very useful for TOPP; it’d be a great way to present the summer’s worth of block parties, for instance. And it’d be fun to add geocoded event support to OpenPlans; users who are producing events (sprints, protests, fundraisers, whatever) would be able to enter the time and location, and their event would show up on a regional timeline map. Other folks in that region would get a great visual overview of what was going on around them.

Beyond just generally learning what was out there, the time spent talking to people and listening to talks left me with a few specific takeaways:

1. Mapping is the new black
   
   Nearly everybody is doing mapping related work. Maybe this was a bit slanted, because “Mash-ups” was one of the themes of the conference and, frankly, I can’t think of a compelling mash-up I’ve seen that wasn’t mapping-related. But I think it’s more than that; mapping is very useful, and people are putting mapping tools to good use in all sorts of ways.
2. TOPP’s Geo group is in a great position
   
   While there are a lot of people doing mapping work out there, TOPP has been doing geo stuff for longer than just about anybody I’ve seen in the open source-y non-profit world. And the depth of the work TOPP has done is impressive; we haven’t just used existing tools in clever ways, we’ve been one of the driving forces behind the very existence of a high-powered open source end-to-end GIS system. I spoke to a couple of folks who had been in consideration for larger geo-related contracts w/ gov’t and NGO organizations where the had really wanted to use the open source offering, but had balked in the end from fear of becoming dependent on a one- or two-person consultant team. With TOPP’s depth of experience and resources (human and otherwise), we’ll be able to make such organizations feel much more comfortable. This has been the plan all along, but it’s really exciting to talk to people who are out there in the middle of it confirming that the opportunities really are there, and that we’re in a great place to be able to take advantage of them.

3. TOPP itself is in a great position
   
   Most of the people who are pouring their lives into using technology for good causes are constantly hustling for funding. Either they’re freelancing and are trying to land their next contract (even as they’re working on their current one), or they’re a non-profit that’s constantly pounding the pavement for grants and other funding. TOPP is very unusual, and very lucky, in that we’re given the opportunity to develop our offerings without a lot of pressure to monetize them; only after we’ve established a reputation and developed demonstrable strengths do we start to focus those strengths towards generating a revenue stream. This seems to be working fabulously for the Geo group (see above); I see no reason why it won’t work similarly for OpenPlans, and other projects that we develop, as long as we continue to produce working, usable software.

I managed to talk to a lot of folks throughout the event, folks were generally very interested in what we’re doing. I had a number of occasions where someone would approach me and introduce themselves, saying “Hi! So-and-so said I should come over and talk to you, that I’d be very interested in what you’re doing. What are you doing?” While it was hard to keep describing our organization and our efforts over and over again without sounding lame and canned, that’s a pretty good problem to have. A couple of the connections I made may end up bearing some fruit down the road. In particular, I’d guess that our Geo folks would be interested in talking to the folks who find out about but do not end up getting those bigger contracts I was mentioning above, so we can decide whether we might be interested in trying to land those jobs. Also, there were a couple of nice mapping front end tools that would probably add value to our greater GIS offerings, if we made sure those tools work with our stack.

All in all I think it was valuable for me to be there. And I think that we should have a slightly bigger presence next year, including giving a presentation of our own. I’m sure they’d be happy to have us.

  This is pretty simple and intuitive I think.  We also have a status of being listed on a project.  That is, can people see you’re part of the team.  If you’re a whistleblower at exxon, you might not want your company to see that you’re part of the ExxonIsSatan project, for instance.  For a particular project, other members can see information about your affiliation according to the following rubrik:

	Listed	Not Listed
anonymous	view	X
site member	view	X
project member	view	view
project admin	view	view

 I’m being intentionally over-verbose here for illustration.  If user A wants to see information concerning member B about his affiliation with project C, how does this work?

- can user A view project C?

- can user A view member B’s affiliation with project C?

So its a cascaded permission system, if you will (or something like that, anyway).  My main reason for posting this is to put this in programmatic terms and to see what others thought about this. We’re not really taking advantage of as much as we can here, and I’m tempted to say that intelligent use of roles could make this all happen behind the curtain, which would be my preference.  Also, why not have a reference on line?

 [Disclaimer:  no exxon executives, stakeholders, or stock prices were harmed either financially or materially in the writing of this post]

• If we’re building on some other code base, use the existing license.
• If we’re building our own code, make something up.

So, sometimes people will ask about some piece of code and its licensing, and I’m forced to kind of pull an answer out of thin air, obviously without a great deal of thought.  It’s nice that we make everything open source — and honestly that’s pretty much all I care about — but we need to be a little more clear about what that means for other people.

So: what licensing should we use?  I wrote a post about my own thoughts on licensing.  In summary, I think the most practical choice is a permissive license on libraries (MIT, BSD, etc), and perhaps the GPL (v3?) on applications.  Though if applications have functionality extracted into libraries then that involves license switching — fine to do internally, but if there’s external contributions it can become complicated.

Ideally, I would like as an end product a document that makes it pretty clear to someone when they start a project (of any size) what how that should be licensed.  I’d rather it not document licensing in terms of options, as I honestly don’t think it’s worth the time to put a lot of thought into licensing on a per-project basis.

Python has a bunch of these frameworks and I think this is a good thing. What I do question is how much functionality lives in the framework that could be abstracted outside of the framework.  What is a framework but a tool kit that you want to apply to HTTP requests and responses?  Thinking that way, issues like handling unicode, HTML escaping, authentication, etc are really library type functions.  If Bob doesn’t like pylons but likes how they do auth, that part should be pretty libraried-out (well, there’s authkit, which is actually a good example of what I consider *bad* library code).

This isn’t a magic bullet, not do I encourage programmers to prematurely make their code library-like.  I’ve been bitten by that too.  But once one figures out process and what one *really* wants to do, its easy to figure out the pattern, figure out which parts are actually *part* of the web framework and which parts are better *consumed* by the web framework.

 So we took out the traversal to find the thread beginning/end and just display the 5 most recent messages regardless of thread.  This brought things down to 5s, but now its different from the design spec of how its displayed.

 Which got me to thinking…

What I was given was a definitive but tentative plan for how the feeds would work.  The blog feed displays the top 5 messages regardless of the most recent comments (the number of which are displayed).  The wiki feed displays the five most recently changed/created pages (there are no “responses” to a wiki page, at least yet).  The listen feed is speced to display the activity of the five most recent threads across all mailing lists (making it a feed aggregator as well as a feed provider).  The team feed displays the members sorted by whether they have a portrait or not, though debate has occurred whether additionally the admins should be displayed on top, or sorting users randomly, or…

It would be nice to have the adaptation controllable with parameters.  How the feed is sorted could be such a parameter, or maybe several parameters if sorting by multiple fields (”i want all admins on top, then members with pictures, with some randomness thrown in”).

I hope we can soon figure out our feed story going forward.  Its come a long way, but still lots left to do.