OpenCore Software

I’m going to be playing Opencore Release Manager, starting now. Let me explain what that means, and relay some plans.

Scope

I’m just talking about opencore, the Plone extension at the heart of the openplans software stack.

I’m not talking about any other part of the openplans software stack (tasktracker, wordpress, deliverance, …)

I’m definitely not talking about openplans.org the website.

Opencore needs a release manager?

Opencore was originally created by TOPP for www.openplans.org. So opencore releases have historically been driven by the needs of www.openplans.org’s deployment schedule. Since our deployment process doesn’t require a formal opencore release, releases tend to be a rare afterthought.

But there are at least two things that more formal releases would help with:

1. Lower the barrier for other community members to get started using opencore.
2. Let the world know about what we’re doing.
   

My overarching goal is to promote a lively community of developers and users for opencore, and provide some balance to TOPP’s path of least resistance–to only worry about our own deployments.

Why me?

Ethan asked me to do it and I was happy to volunteer. (If you don’t know who Ethan is either, this picture should suffice: http://www.openplans.org/people/ejucovy )

Who?

I’m Paul Winkler, a Zope user and developer since 1999, a TOPP employee since October 2007, and Opencore’s release manager since right now. http://www.openplans.org/people/slinkp

What will determine the release schedule?

I want to decouple Opencore’s release cycle from the TOPP deployment cycle. Douglas Mayle has volunteered to act as the openplans.org deployment manager; he and I have tentatively agreed on this plan:

• Opencore releases will be on a time-based schedule, TBD.
• Features that aren’t ready in time for a release can just wait until the next release.

  Rationale: Feature-driven releases inevitably get delayed and delayed, and the features don’t get done any faster.

• Openplans.org deployments will be on an independent time-based schedule. (I believe the plan is to deploy weekly.)
• “Must-have” openplans.org features and fixes can be deployed independently of the opencore release cycle by putting them in a new package that layers on top of opencore.
• Openplans-specific code and text (like the current copyright notice) will be gradually moved out into this new package.
• The aforementioned openplans.org customization package does not exist yet; we need to create one ASAP.
  

Of course TOPP’s own needs will continue to drive a lot of opencore development, which is a good thing; but that shouldn’t be a burden on you, the wider opencore user/developer community.

Does the Release Manager have to be a TOPP employee?

No. I’m just the first. It should be a rotating job. It’s my hope that before long, people outside TOPP will be willing and able to play this role.

When will the job rotate?

Don’t know yet.

What does the Opencore Release Manager do?

• Manage the release schedule (not TOPP’s deployment schedule!). This does not mean I’m the dictator, just the point man and coordinator.
• Shepherd release candidates through a stabilization process.
• Create and announce final release packages.
• Document upgrade process.
• Document any backward incompatibilities.
• Draft and enforce relevant dev process standards (e.g. which branches or plugins should you commit bugfixes or features to).
• Document this process for the next release manager.

At this stage, I’ll also be acting as a sort of ambassador between TOPP and the wider community. Which is a bit silly, as every opencore contributor does that to whatever extent they wish. But I think for now it’s useful to consciously formalize it as part of my job. I will make an effort to:

• Solicit community input about decisions we need to make for opencore.
• Remind TOPP employees to discuss and make decisions about opencore out in the open. If you have a real-life relevant conversation I’ll try to remind you to post a summary to the mailing list.
• Keep the community updated on changes to our build process. If we break your builds, you can complain to the opencore-user or opencore-dev mailing list, and you should expect a response from me.
• Lobbyist for the community: e.g. prioritizing and scheduling feature requests and bugfixes.

There will of course be other forces at TOPP driving things with other priorities, but *my* role will be to ensure that the community’s needs don’t get ignored. This is more about release management than development management: I’m not in charge of deciding which features get done when. But whenever good code comes from the community, we have a responsibility to get that code released in a timely manner, and generally be as responsive as we can.

What about the rest of the stack?

We realize that most of the non-TOPP users of opencore are using some or all of the entire www.openplans.org stack. We’d eventually like to be making formal releases of all the packages, and possibly some kind of “batteries included” big meta-package; but that’s in the future. For now, opencore is our fastest-moving target and the most in need of release management.

Release Numbers

We supposedly have a policy now (Ethan proposed it here: http://tinyurl.com/4ybpro) … but we haven’t been sticking with it. I’m going to draft a proposal for a simpler convention and process. More on this in a later message.

Releases, Eggs, and Builds

I’d like to be releasing versioned eggs of opencore to PyPi. These should be installable into a suitable Zope/Plone instance using nothing more than easy-install.

(We have made a few releases to PyPi, but they aren’t actually usable … some files are missing.)

For bootstrapping a full stack including Zope, Plone, and other non-egg stuff, TOPP will continue to use and develop our Fassembler build tools. But it should become easier to do without that, if you want.

Branch Policy

We are currently trying to follow a “Stable Trunk” practice. Experimental and risky code should not be on the trunk.

Developer Infrastructure

This is only tangentially related to release management. I just wanted to note in passing that we’re planning a more thought-out integrated home on the web for opencore development. We’re calling this the “TOPP Dev Center” for now. Other open-source packages from TOPP will live there too.

In the meantime:

Wiki and mailing lists are at: http://www.openplans.org/projects/opencore

Trac (bug tracking) is at: http://trac.openplans.org/openplans

If you need a Trac account, just ask. (I wish we could take anonymous bug reports, but we got trac spam. There is a guest account, but I’m not sure of the login details.)

If you want SVN commit access, first you should send some bugfixes and features as patches (either in trac or on the dev list). We’ll happily give commit access to people who have a history of submitting good patches. There will be a contributor agreement you’ll have to sign, somewhat similar to Zope’s.

Current Road Map

Some things coming up in the short term:

• Plone 3

  We will soon be releasing a version of opencore that runs on Plone 3. Rob Miller has done all the hard work and we just need to QA this branch and merge it back to the trunk.

  Currently, the plan is:
  

  1. TOPP will create a release candidate from the plone3 branch.
     
  2. TOPP will QA this branch.
  3. TOPP will deploy this branch to www.openplans.org.
     
     When that goes well, I will:
  4. Branch off the current opencore trunk to an opencore-plone2.5 branch
  5. Make a final stable release from the opencore-plone2.5 branch (to be followed by bugfix releases as necessary)
  6. Merge the plone 3 branch back to the trunk.

  This will definitely happen this summer; hopefully even in July. More concrete dates will be forthcoming.

  If people in the community expect to continue using the Plone 2.5 version for a while, I would like to hear about it so I can serve you better.

• Easier “plugin” management using z3c.autoinclude. This will replace our usage of zcmlloader, doing the same job but better (and other people are actively developing it).
  

Some ongoing maintenance tasks I’d love community help with:

• Opencore is slow. Let’s make things faster. (But let’s get plone 3 merged to trunk first, it already helps substantially.)
• The test suite has a lot of problems. More on this in a later message.
• Dependency cleanup. I suspect there is cruft in the Products “bundle” that we don’t actually use.
• What do you guys want?

Going forward, this road map should have a permanent home prominently linked in the aformentioned dev center. There is already such a page at http://www.openplans.org/projects/opencore/planning but it badly needs an update. I’ll be looking to revive that.

Questions?

Whew. Sorry for the length. Feedback would be most welcome.

- PW

This past Tuesday I attended the NYC Python User’s Group meeting at the offices of DayLife.

The presentation this week was by Peter Fein about GrassyKnoll, a text search engine written in Python.

From TOPP’s point of view there are several interesting things about it:

• It could provide us with a sitewide search solution for openplans.org and livablestreets.com.
  
• It can use any of a number of pluggable back ends (notably PyLucene).
  
• You interact with it entirely via a simple REST API.
  
• A GrassyKnoll client can also trivially be a GrassyKnoll server. (Peter gave a live demo of this.) In theory this could allow for fun things like smart clustering, where one server gets your query and dispatches queries to N other servers, and then merges the results appropriately. The end client still sees the same simple rest API.
• It’s written entirely in Python and is Free.
• They’re looking for more people to help out and would love for us to get involved.

The major down side is that they’re still some months away from a production-ready release. (There were a few glitches in the live demo.) When I pressed him about this, Peter said “definitely by the end of the year.” I got the impression that more hands helping would speed that up.

Also, it doesn’t define any kind of common query language; it just passes them along to the back end. So you do have to know what you’re really talking to.

Peter said he was hopefully coming to the next NYCPUG meeting that we’re hosting here at TOPP on July 15.

I would like us to figure out our licensing policy.  So far the policy has been:

• If we’re building on some other code base, use the existing license.
• If we’re building our own code, make something up.

So, sometimes people will ask about some piece of code and its licensing, and I’m forced to kind of pull an answer out of thin air, obviously without a great deal of thought.  It’s nice that we make everything open source — and honestly that’s pretty much all I care about — but we need to be a little more clear about what that means for other people.

So: what licensing should we use?  I wrote a post about my own thoughts on licensing.  In summary, I think the most practical choice is a permissive license on libraries (MIT, BSD, etc), and perhaps the GPL (v3?) on applications.  Though if applications have functionality extracted into libraries then that involves license switching — fine to do internally, but if there’s external contributions it can become complicated.

Ideally, I would like as an end product a document that makes it pretty clear to someone when they start a project (of any size) what how that should be licensed.  I’d rather it not document licensing in terms of options, as I honestly don’t think it’s worth the time to put a lot of thought into licensing on a per-project basis.

Last night Google announced Google App Engine.  Since then I’ve been pretty obsessed with it.  (If you are interested in trying it out, you should sign up and put in a request — they seem to be sending out invites periodically).

Since last night I’ve been reading about it, and some of the commentary around it.  I haven’t actually tried to run anything on it yet.  But here’s my first impressions:

It’s not as peculiar system as some people have suggested.  It runs CGI-ish scripts, and it is easy to turn that into WSGI.  Processes can be reused, but they are short lived and single-threaded.  You can’t have any background processes.  It’s a lot like the PHP process model.  An SDK is provided that allows you to run stuff locally.  The SDK is open source (Apache licensed), so at least a minimal environment is available with no proprietary ties, and it provides a model for reimplementing proprietary parts of the stack.

There’s no database, but there is Google BigTable stuff, with a database-like API.  It accepts queries that look like SQL, and they give a Django-like ORM.  It’s close enough to be familiar, but it’s not exactly a database.  Like the ZODB, you have to add indexes for any queries you want to run outside of the most obvious queries.  It might be more accurate to look at BigTable as ZODB-like than RDBMS-like.  But I don’t know; it’s a big topic and I’ve only read that part of the docs lightly.  This is probably the biggest proprietary tie that an application written for appengine will have.  But I think that making this API work over the ZODB would be feasible.  You won’t get the same scaling properties of BigTable, but the application would still be viable.  There’s also some APIs for email, authentication, and doing URL requests.  Only authentication really seems concerning from the perspective of tie-in.  Google, unfortunately, has not been particularly progressive with respects to OpenID.  But people have already used OpenID with this, so it’s not a hard constraint.  Given Google’s involvement in Open Social stuff, I also imagine that their stance on authentication will improve.

That’s just a short description, but I post it here because I’ve been thinking about what if anything this means for TOPP development.

Generally, I think this could be a very big thing for Python web development.  This offers a development environment that is on par with PHP (which I think has a very good and accessible development environment, IMHO the most significant reason for its success).  Well, better than PHP really, as you get most of the deployment advantages, but they’ve structured the system in a way to make good development practices easy (private staging deployments, you can update and revert versions of an application, and probably other stuff I haven’t yet noticed).

This relates to us because of what this could do to the general ecosystem for open source web applications.  Right now deployment is hard.  Hard enough to seriously stunt the success of open source web application projects.  Even the most successful applications — for example, Trac and MediaWiki — seem to be a flash of activity until they are superseded by easier-to-manage hosted services.  Google Code’s issue tracker is a far cry from Trac, and not extensible, but it’s so much easier to manage that I have a hard time recommending going through the trouble of setting up Trac (and dealing with spam and other problems) when that time could be better spent actually writing code.  Supporting deployment on systems also is a lot of overhead for open source developers, and the overhead has very little return.  Supporting more deployment options doesn’t actually improve a product.

So as a result the emphasis has been on hosted services, and open source development effort has been focused more on tools for building those services than on the services themselves.  In some ways this seems reasonable: with public, free, hosted services (which is a lot of services these days) why do you need more than one implementation?  But of course that’s a simplification, and there are many reasons you might want to modify a web application at the code level.  And while mashups offer some extensibility without code sharing, and with closed/hosted services, they can be quite limited.  Unless you write your own applications, which is too difficult for many people to do reliably because deployment is hard, expensive, and hard to scale in response to the surge in traffic a successful mashup can get (a surge which might not turn into any means of economic stability, as the surges themselves are unstable).

Google App Engine could change this.  This is a hosted platform that appears that it will, when it gets out of beta, offer basically free hosting to small web applications.  (The quota limits, which they say will remain for the free service after it gets out of beta, are quite generous.)  So, for instance, if the Trac developers make a port to this environment (which seems quite possible) then installing Trac, with whatever plugins you want, and any local modifications you care to make, suddenly seems very feasible.  Even feasible for people who could only marginally be classified as “developers” — a class of people who have almost entirely gone to PHP up until now.

Internally we’ve always struggled with how functionally open source our products can be.  That is, while people are allowed to use the stuff we write (via licensing), that does not necessarily make the stuff we write compelling or useful.  Deploying our entire stack is somewhat challenging, and even putting aside the technical points of that deployment, there’s the concern about whether it is a useful basis for someone outside of our team to make a site.  This might be resolvable, but when open source web applications have such a limited potential for success it makes it hard to justify resolving problems.  This could substantially increase the motivation to create reusable applications.

We’re at PyCon! The lion’s share of the openplans / opencore developers are here:

Ian Bicking, Josh Bronson,  Mel Chua, Jeff Hammel, Anil Makhijani, Robert Marianski, Doug Mayle, Whit Morris, and me (Paul Winkler). Look for us and say hi!  We are friendly and mostly harmless. Oh, and we might have a job or two open.

I’m giving a talk Friday at 3:55, and Ian’s doing one on Sunday at 11 AM

A lot of times at the Snow Sprint I was struck by a feeling that I was in the middle of a group of people who were really doing open source right. So on the plane home I started thinking about what that meant and why I haven’t felt that way at TOPP.

Well, a lot of us have said at various times that our attempts at doing open source development have often been frustrating and rarely seem to be worth the effort. Coming out of the sprint, I think that’s because we are missing a very big piece of the “True Spirit of Open Source”. Doing open source properly is not just a matter of putting the GPL on our code—it is not about making a legal contract but about making a social one. It is about committing to reuse and collaboration both internally and externally and it is well worth the effort.

Writing, finding, and using software all take time and care and with someone paying for our time and care we ought to think of these acts as investments. We should make the most of those investments by seeking code reuse and encouraging collaboration. The more we encourage heavy use of software and heavy reuse of code, the better our ability to improve and solidify it. Whether or not we’ve read Dreaming in Code (though everyone here should—as soon as possible—seriously) we all know that software is hard to do well. It’s full of unexpected behavior, prone to breaking when you touch it, and rarely optimized. We can improve it only by testing and exercising it: give it to people to use, poke it in new ways, put it under stress, see what breaks, fix it and don’t let it break in the same way again.

Laziness is a virtue

Because software is expensive to write, it is most responsible of us to write as little as we can. And because software is hard to write, it’s rarely done once it’s written; most software has tremendous room, and often necessity, for improvement. So our goal should be to maximize the impact of every improvement we make. This primarily means seeking to increase the breadth of our code’s effect: convincing other people to use our software, sharing code across multiple products, and reusing products in as many ways as we can across everything we develop.

To give a specific, if vacuous, example, we have five or more different pieces of code deployed on OpenPlans that generate and send email to users: Listen sends mail, Twirlip sends mail, WordPress sends mail, and Opencore sends mail in at least two different ways. Suppose, instead, that we had a single shared way to send mail to users. Aside from the immediate benefit of less overall knowledge to store, the different mail-sending needs of all these applications would force us to develop a stable and reliable method for sending mail, and when heavy usage of one of those dependent applications inevitably reveals an inefficiency or bug in the mail-sending code, multiple otherwise unconnected parts of our site would instantly benefit from our resulting improvements. Put that same mail-sending code on two or more websites and encourage other developers to deploy it in their applications, and we can tremendously increase both the number and variety of stress points and the magnitude of impact of each strengthening, both the opportunity and the payoff for improvements.

So maximizing adoption of every piece of software by both users and developers is a forward-thinking and cost-effective stance. It gives programmers the most opportunities to strengthen their (buggy, brittle) code and ensures that every (hard, pricey) act of improvement has a large and wide-reaching effect.

We’re not alone

It is not only creating software that needs be treated as an investment. In a true open source environment using others’ software, too, is an investment that we should make the most of. When we choose to depend upon existing free technology—in any way: WordPress, Xinha, Zope, Pylons, Trac—we will benefit ourselves and others by doing all that we can to improve it, participate in its communities of users and developers, and encourage the growth of those communities. The core reasons are the same as with our own code: the more eyes and hands that are on a piece of software, the better it can become.

There is, of course, an additional benefit to encouraging outside developers, namely lessening our internal development and maintenance burden by gaining volunteer contributors. When we create our own software we thus have the additional responsibility of building these communities from nothing. But when we are dealing with existing open source software this goal is all the easier to achieve. A community of developers already exists; our only responsibility is to gain admittance. We should treat our lack of commit rights not as an obstacle, but as a challenge.

Alongside the fuzzy feelings that come with earning the respect of an established group, there is, in keeping with the best of the open source mentality, great potential for everybody to win when we are welcomed into an existing community. We stand to reap the benefits of the efforts of the intelligent, experienced and caring people who develop the software that we have decided is worthy of our use, and we gain the potential to influence the path and priorities of their development with our input and interest. They, meanwhile, gain the feedback and work of a new set of smart (and funded) contributors with a real interest in the success of their project.

Finding those unicorns

I’ve framed this primarily as an economic argument to be concrete, but of course there’s more to it than that. The same stance is also about committing to respectful and positive relationships with outside agents; putting our faith in open and collaborative development environments and in the potential for groups of smart, committed people to produce great things even with differing agendas; taking the time and care to make our work as good as it can be and letting the rest of the world be the judge of it. It’s about being responsible consumers who contribute back to the products we consume, and being responsible producers who proudly stand by what we produce. And then there’s our aspirations at TOPP to do good and produce real positive change: all the time and money we can save by reusing and building on existing solutions can be diverted toward new projects to do even more good in the world, with cascading effects.

So let’s build and get involved with communities, submit and fix bugs in others’ code, give thanks and contribute ideas to the people whose software we use.

Let’s release Flunc, Externalator and Deliverance to the world for real, with welcoming websites and helpful documentation, and encourage people to use them and talk to us about them on mailing lists and IRC channels. Along with everything else it’s a good way to put our software to the test: to put a different spin on the dogfood concept, if no one else wants to use our software, why should we? Let’s keep up with Xinha development and submit behavioral patches to its authors instead of keeping our work isolated on our “vendor branch.” Let’s tell people about the recommendations of our smart interaction designers and submit our implementations of that advice; how many open source software communities have an excess of input from web software designers? Let’s release WordPress and Trac plugins to their wider communities and let’s solicit contributions and feedback to Listen and Opencore.

In short, let’s try to practice better open source. After all, we can only bring the open source ethos and philosophy to the broader world after we get it right in our software.