OpenCore Software

Should I upgrade?

This is a bugfix-only release. If you are running opencore 0.13 or 0.14, this should be a safe upgrade with a few useful fixes.

If you are still running opencore 0.12 or earlier on top of Plone 2.5, this is a fairly big upgrade; see the 0.13 announcement for more info.

What’s New Since 0.14.0

Data Migrations: You will need to run this migration only if you’re upgrading from 0.13 or earlier. You can run the migration via the portal_setup tool in the ZMI; go to the upgrades tab and run the upgrade registered for version 0.14.0.

Bug fixes:

  * List manager address was showing up as escaped HTML.  (slinkp) (http://trac.openplans.org/openplans/ticket/2706)

 * Wiki version view doesn’t error when no version_id is specified  (slinkp) (http://trac.openplans.org/errors-openplans/ticket/30)

 * Error page doesn’t barf on 404s if the catalog has a problem coming up with suggestions based on the current URL (slinkp)  (http://trac.openplans.org/errors-openplans/ticket/23)

 * Fixed rare error in recataloging projects, raised by the project’s _default_img_data method. (slinkp)

There are also a bunch of upstream bugfixes in Listen, our mailing list component. You will want to upgrade if you’ve had any trouble with non-ASCII characters in mail messages.

How to Install

As usual, I recommend building using our usual build process.  Whether you use the newbuild.sh wrapper script from the fassembler-boot package, or just use fassembler directly, you have a choice of two requirements sets:

• opencore-minimal/tags/0.14.1 is a requirements profile for a fairly stripped down opencore build. It includes supervisord and zeo, but does not include tasktracker, wordpress, or deliverance.
  
• openplans/releases/opencore-0.14.1-rc builds a more complete stack including tasktracker and wordpress blogs; this is equivalent to what we’d deploy on www.openplans.org.   We haven’t fully vetted this build for deployment yet, but it passes all functional and unit tests.
  

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

What took so long?

I realize it’s been over a month since Opencore 0.14.0.  One thing that added to the time was that, this time I was very careful to create a frozen list of all python packages that opencore depends on, directly or indirectly, so that building this release should be reliable and repeatable. No more depending on whatever happens to currently be in SVN.  And since I’d never done this for 0.13 or 0.14.0, it took longer and I ran into more problems than I expect to for future bugfix releases.

Pip was very useful for this job, although I had to massage the output a little bit to make fassembler happy.

I also took the time to edit my notes and document exactly how I make a release.  I’m using this as a checklist for now.  In the future I’d like to automate some of the more mechanical parts of the process.

Opencore, the software that powers much of www.openplans.org, www.livablestreets.com, www.openesf.net and others, has a new release available today.

If you did not upgrade to 0.13, I recommend skipping 0.13 and going directly to 0.14.

Features

This is mostly a bugfix release. There are a couple of very small features (all courtesy of David Turner), notable to integrators rather than end users:

• search engine optimization tweaks
• Added per-project security context page for twirlip and other uses at projects/foo/security-context
• Add twirlip transclusion link for wiki pages.
  

Bug Fixes

Lots! A partial list:

• private project team pages didn’t always show all members
• Several performance improvements
• Wiki history “Last edit performed by…” now correctly shows the last edit, not the first
• Wiki history comparison errors fixed
• Lots of Unicode errors in project joining and inviting fixed
• NameError in project joining fixed
• Errors fixed when deleting pages or images from the project contents view
  
• The scripts to migrate from version 0.12 or earlier got some fixes to correctly update all mailing lists

For complete history, see CHANGES.txt in the opencore source.

How to Upgrade

If you are upgrading from opencore 0.12 or earlier, you must carefully follow the 0.13 upgrade and migration instructions. Be aware that such an upgrade requires a data migration that is not backward compatible! See my blog post about the 0.13 release for details.

I also recommend upgrading Listen to at least revision 72388 to get a bunch of Unicode decode / encode error fixes, a fix that makes the list footer show up correctly on multipart messages, and other goodness. An appropriate version can be found in our big tarball of plone products.

Here are general instructions on how we at the Open Planning Project do builds and upgrades. We don’t yet have a requirements profile that uses the 0.14.0 tag, but a requirements profile for a minimal opencore build using the (currently identical) 0.14 branch is here: https://svn.openplans.org/svn/build/requirements/opencore-minimal/branches/opencore-0.14

After Upgrading: Migrations

There is one migration you should run to be sure that team views show all the team members. From the command line, run this zopectl script:

zopectl run  opencore/migrations/update_member_project_ids.py

It may take a while if you have lots of members. (A test run with 3700 member accounts took 6 minutes.) But if you’re using ZEO, you can safely run it after restarting Zope–no downtime is required.

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

What’s New in Opencore 0.13

The biggest changes in this release are:

• Plone 3 required!
  
• Many things are noticeably faster, because Plone 3 is faster than Plone 2.5.
• Lots of minor bugfixes and usability improvements

For more details, see CHANGES.txt in the opencore source.

This release also requires a data migration that is not backward compatible! Let me say that again:

This release requires you to run a data migration that is not backward-compatible.

Once you upgrade, you can’t go back.

Always Make Backups!

Ahem.

Also, if you have your own code that runs on top of opencore, you should plan on doing thorough testing and may need to modify your code to run under Plone 3.

How to Upgrade

First, make a backup of your ZODB.

Next, run a pre-upgrade migration as described here:

https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

Note that you need to be on Opencore 0.12.1 (or later, if we ever do an 0.12.2 etc.) to have that migration script available. So if you’re running an earlier version, you’ll need to upgrade to 0.12.1 first.

Then you’re ready to upgrade to 0.13. For this upgrade, I highly recommend leaving your old build alone in case something goes wrong, and create a new build from scratch in a parallel directory. As with previous releases, Opencore still doesn’t work via easy_install, you have to get it by checking out the tag from subversion: https://svn.openplans.org/svn/opencore/tags/0.13.0 and then install using python setup.py develop. You’d also have to install Zope 2.10 and our plone 3 Products tarball. Or use TOPP’s usual build tools which will do all that for you.

I highly recommend the latter approach, it’s a lot more convenient. We have requirements profiles ready for you to build:

• For a pretty stripped down opencore build (just Opencore, Zope, Zeo, and supervisor; no Tasktracker, no Wordpress, etc.), you can use these requirements: https://svn.openplans.org/svn/build/requirements/opencore-minimal/tags/opencore-0.13.0
• For a full-featured build like openplans.org, with wordpress and tasktracker and everything, you can use these requirements: https://svn.openplans.org/svn/build/requirements/openplans/branches/opencore-0.13
  
  … although I’m not sure that’s the right place for that requirements set; Doug Mayle, the deployment manager, could overrule me and put it somewhere else. I’d suggest keeping an eye on https://svn.openplans.org/svn/build/requirements/releases where the final requirements are likely to show up once Doug does all his work to prepare for deploying to openplans.org. That would also be a good sign that we’re getting confident enough to deploy it ourselves

After you upgrade, you’ll need to run the post-upgrade migrations. Again, see:

https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

Start things up and you should be all set!

Special Thanks

The Plone 3 porting was almost entirely performed by Rob Miller. It was a big job, thanks Rob for moving us forward!

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

I will be on vacation until 2008/08/25. Good luck while I’m gone

– Paul Winkler

Hi folks, opencore 0.12.1 was released today.

This is a bugfix release with no user-visible new features.

As with the previous release, it still doesn’t work via easy_install, you have to get it by checking out the tag from subversion: https://svn.openplans.org/svn/opencore/tags/0.12.1 and then install using python setup.py develop. Or use TOPP’s usual build tools which will do that for you; however, we don’t yet have a requirements profile ready that uses this tag. You could modify requirements/opencore-req.txt if you have an existing fassembler build that you want to upgrade.

The biggest changes in this release are:

• Force project content deletion to be done via POST; if it’s a GET, redirect to a confirmation form. Reduces chance of accidental content delete by non-javascript clients. Thanks to Alex Clark for being so helpful and patient with us on this one.
• Added migrations/unmake-sites.py script to unmake the local sites at the app root and the Plone site root. Run this JUST BEFORE updating to a Plone 3-based version of opencore, which will be the next feature release, version 0.13. Do not run this migration under any other circumstances or you will break your site! For more info, see: https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

I will be on vacation until 2008/07/31, I’m hoping my fellow TOPP employees can provide support while I’m gone.

– Paul Winkler

Get it from https://svn.openplans.org/svn/opencore/tags/0.12.0

I am not uploading a package to Pypi because we still have some work to do before opencore works as an egg. I expect to be working on this over the next few releases. For now, the most reliable way to install is still setup.py develop (or use Fassembler which does it for you).

This is as good a time as any to announce that I hope to release about once a week for the rest of the summer! I will miss a few due to vacations. But hopefully 0.13 will be released by July 21.

What’s new in 0.12.0

This is a summary of user-facing changes. For a complete list, see CHANGES.txt in the source code.

* New form for deleting accounts. This is not linked to from anywhere yet, but if you need to delete a spammer, or if a user wants to leave the site, point your browser at people//delete and fill out the form.

* Zip file export view of project wiki pages. This is not linked to from anywhere yet; you have to visit projects//project-wiki-export.zip. Page attachments are not included.

* Email obfuscation for discussion list views

* fix team AttributeError reported by user on summary page

* fix site_url AttributeError reported by user on password reset

* fix user reported UnboundLocalError on pending page

* possibly resolve user reported error on attachment delete

A decree from the release manager:

Until further notice, we will use the common three-component major.minor.patch version numbering scheme, as popularized by Apache and many other projects.

• Major releases (e.g. 1.0.0, 2.0.0) indicate major featureset milestones or large backward incompatibilities.

  Our major release number will remain 0 until further notice (i.e. until we figure out WTF a 1.0 release would mean).

• Minor releases (e.g. 0.12.0, 0.13.0) indicate new features.

  Minor releases may also require data migrations and will generally also include all the bugfixes since the previous minor release.

• Patch (aka “point” or “micro”) releases (e.g. 0.12.1, 0.12.2) may only contain bugfixes, and must be 100% backward compatible.

  No new features are allowed in a patch release. No data migrations may be required by a patch release except as strictly required for a critical bugfix, in which case I might decide to bump the minor version anyway.

You might notice with this policy that minor numbers will get bumped quite often. That’s okay. There’s nothing wrong with numbering a release eg. 0.87.0, except that it would strongly suggest a need to re-evaluate the 1.0 milestone

This is basically the same numbering scheme that we’ve had since we broke out of the infinite 0.9.7.x loop. The main difference is organizational - one person (me) will be paying attention to keeping it consistent. That was the only real problem with Ethan’s previous plan.

I’m going to be playing Opencore Release Manager, starting now. Let me explain what that means, and relay some plans.

Scope

I’m just talking about opencore, the Plone extension at the heart of the openplans software stack.

I’m not talking about any other part of the openplans software stack (tasktracker, wordpress, deliverance, …)

I’m definitely not talking about openplans.org the website.

Opencore needs a release manager?

Opencore was originally created by TOPP for www.openplans.org. So opencore releases have historically been driven by the needs of www.openplans.org’s deployment schedule. Since our deployment process doesn’t require a formal opencore release, releases tend to be a rare afterthought.

But there are at least two things that more formal releases would help with:

1. Lower the barrier for other community members to get started using opencore.
2. Let the world know about what we’re doing.
   

My overarching goal is to promote a lively community of developers and users for opencore, and provide some balance to TOPP’s path of least resistance–to only worry about our own deployments.

Why me?

Ethan asked me to do it and I was happy to volunteer. (If you don’t know who Ethan is either, this picture should suffice: http://www.openplans.org/people/ejucovy )

Who?

I’m Paul Winkler, a Zope user and developer since 1999, a TOPP employee since October 2007, and Opencore’s release manager since right now. http://www.openplans.org/people/slinkp

What will determine the release schedule?

I want to decouple Opencore’s release cycle from the TOPP deployment cycle. Douglas Mayle has volunteered to act as the openplans.org deployment manager; he and I have tentatively agreed on this plan:

• Opencore releases will be on a time-based schedule, TBD.
• Features that aren’t ready in time for a release can just wait until the next release.

  Rationale: Feature-driven releases inevitably get delayed and delayed, and the features don’t get done any faster.

• Openplans.org deployments will be on an independent time-based schedule. (I believe the plan is to deploy weekly.)
• “Must-have” openplans.org features and fixes can be deployed independently of the opencore release cycle by putting them in a new package that layers on top of opencore.
• Openplans-specific code and text (like the current copyright notice) will be gradually moved out into this new package.
• The aforementioned openplans.org customization package does not exist yet; we need to create one ASAP.
  

Of course TOPP’s own needs will continue to drive a lot of opencore development, which is a good thing; but that shouldn’t be a burden on you, the wider opencore user/developer community.

Does the Release Manager have to be a TOPP employee?

No. I’m just the first. It should be a rotating job. It’s my hope that before long, people outside TOPP will be willing and able to play this role.

When will the job rotate?

Don’t know yet.

What does the Opencore Release Manager do?

• Manage the release schedule (not TOPP’s deployment schedule!). This does not mean I’m the dictator, just the point man and coordinator.
• Shepherd release candidates through a stabilization process.
• Create and announce final release packages.
• Document upgrade process.
• Document any backward incompatibilities.
• Draft and enforce relevant dev process standards (e.g. which branches or plugins should you commit bugfixes or features to).
• Document this process for the next release manager.

At this stage, I’ll also be acting as a sort of ambassador between TOPP and the wider community. Which is a bit silly, as every opencore contributor does that to whatever extent they wish. But I think for now it’s useful to consciously formalize it as part of my job. I will make an effort to:

• Solicit community input about decisions we need to make for opencore.
• Remind TOPP employees to discuss and make decisions about opencore out in the open. If you have a real-life relevant conversation I’ll try to remind you to post a summary to the mailing list.
• Keep the community updated on changes to our build process. If we break your builds, you can complain to the opencore-user or opencore-dev mailing list, and you should expect a response from me.
• Lobbyist for the community: e.g. prioritizing and scheduling feature requests and bugfixes.

There will of course be other forces at TOPP driving things with other priorities, but *my* role will be to ensure that the community’s needs don’t get ignored. This is more about release management than development management: I’m not in charge of deciding which features get done when. But whenever good code comes from the community, we have a responsibility to get that code released in a timely manner, and generally be as responsive as we can.

What about the rest of the stack?

We realize that most of the non-TOPP users of opencore are using some or all of the entire www.openplans.org stack. We’d eventually like to be making formal releases of all the packages, and possibly some kind of “batteries included” big meta-package; but that’s in the future. For now, opencore is our fastest-moving target and the most in need of release management.

Release Numbers

We supposedly have a policy now (Ethan proposed it here: http://tinyurl.com/4ybpro) … but we haven’t been sticking with it. I’m going to draft a proposal for a simpler convention and process. More on this in a later message.

Releases, Eggs, and Builds

I’d like to be releasing versioned eggs of opencore to PyPi. These should be installable into a suitable Zope/Plone instance using nothing more than easy-install.

(We have made a few releases to PyPi, but they aren’t actually usable … some files are missing.)

For bootstrapping a full stack including Zope, Plone, and other non-egg stuff, TOPP will continue to use and develop our Fassembler build tools. But it should become easier to do without that, if you want.

Branch Policy

We are currently trying to follow a “Stable Trunk” practice. Experimental and risky code should not be on the trunk.

Developer Infrastructure

This is only tangentially related to release management. I just wanted to note in passing that we’re planning a more thought-out integrated home on the web for opencore development. We’re calling this the “TOPP Dev Center” for now. Other open-source packages from TOPP will live there too.

In the meantime:

Wiki and mailing lists are at: http://www.openplans.org/projects/opencore

Trac (bug tracking) is at: http://trac.openplans.org/openplans

If you need a Trac account, just ask. (I wish we could take anonymous bug reports, but we got trac spam. There is a guest account, but I’m not sure of the login details.)

If you want SVN commit access, first you should send some bugfixes and features as patches (either in trac or on the dev list). We’ll happily give commit access to people who have a history of submitting good patches. There will be a contributor agreement you’ll have to sign, somewhat similar to Zope’s.

Current Road Map

Some things coming up in the short term:

• Plone 3

  We will soon be releasing a version of opencore that runs on Plone 3. Rob Miller has done all the hard work and we just need to QA this branch and merge it back to the trunk.

  Currently, the plan is:
  

  1. TOPP will create a release candidate from the plone3 branch.
     
  2. TOPP will QA this branch.
  3. TOPP will deploy this branch to www.openplans.org.
     
     When that goes well, I will:
  4. Branch off the current opencore trunk to an opencore-plone2.5 branch
  5. Make a final stable release from the opencore-plone2.5 branch (to be followed by bugfix releases as necessary)
  6. Merge the plone 3 branch back to the trunk.

  This will definitely happen this summer; hopefully even in July. More concrete dates will be forthcoming.

  If people in the community expect to continue using the Plone 2.5 version for a while, I would like to hear about it so I can serve you better.

• Easier “plugin” management using z3c.autoinclude. This will replace our usage of zcmlloader, doing the same job but better (and other people are actively developing it).
  

Some ongoing maintenance tasks I’d love community help with:

• Opencore is slow. Let’s make things faster. (But let’s get plone 3 merged to trunk first, it already helps substantially.)
• The test suite has a lot of problems. More on this in a later message.
• Dependency cleanup. I suspect there is cruft in the Products “bundle” that we don’t actually use.
• What do you guys want?

Going forward, this road map should have a permanent home prominently linked in the aformentioned dev center. There is already such a page at http://www.openplans.org/projects/opencore/planning but it badly needs an update. I’ll be looking to revive that.

Questions?

Whew. Sorry for the length. Feedback would be most welcome.

- PW

Rob Miller has been thinking about zc.buildout lately, and providing a buildout-based bootstrap for our stack (even if it only calls out immediately to fassembler).  This got me to thinking, and I finished a reflection post about fassembler.

At the end I almost talked myself into refactoring stuff to make the good parts of fassembler usable in zc.buildout, so that we could move over to that in the future if we want to.  Then… right at the end… pow!  Files!  All files, only files!

What if our build process just built files?  Relocatable files.  Not-system-dependent files.  You could tar the whole thing up, copy it to another location or server, unpack it, and get the exact same site.  That tarball would basically be equivalent to the build itself.

It’s not that App Engine seems particularly awesome.  More that it seems like it might be Just The Right Kind Of Dumb.  It’s that this makes it so easy to think about the build, about what it does, about stuff like the workflow for testing and deployment and rolling back.  Easy to explain to other people, easy to audit, easy to debug.

The problem of course is that it requires pushback into other products.  Lots of other products.  But we can do that.

Thoughts?

This past Tuesday I attended the NYC Python User’s Group meeting at the offices of DayLife.

The presentation this week was by Peter Fein about GrassyKnoll, a text search engine written in Python.

From TOPP’s point of view there are several interesting things about it:

• It could provide us with a sitewide search solution for openplans.org and livablestreets.com.
  
• It can use any of a number of pluggable back ends (notably PyLucene).
  
• You interact with it entirely via a simple REST API.
  
• A GrassyKnoll client can also trivially be a GrassyKnoll server. (Peter gave a live demo of this.) In theory this could allow for fun things like smart clustering, where one server gets your query and dispatches queries to N other servers, and then merges the results appropriately. The end client still sees the same simple rest API.
• It’s written entirely in Python and is Free.
• They’re looking for more people to help out and would love for us to get involved.

The major down side is that they’re still some months away from a production-ready release. (There were a few glitches in the live demo.) When I pressed him about this, Peter said “definitely by the end of the year.” I got the impression that more hands helping would speed that up.

Also, it doesn’t define any kind of common query language; it just passes them along to the back end. So you do have to know what you’re really talking to.

Peter said he was hopefully coming to the next NYCPUG meeting that we’re hosting here at TOPP on July 15.

So, Zope’s handling of XML page templates is kind of broken.

As a result, all of opencore’s page templates are parsed in ZPT’s “html mode”. This makes for some surprising validation behavior (e.g. unclosed tags is not an error, but putting content in a “link” tag is an error… which sucks if you happen to be trying to generate georss and not HTML …)

I ended up not even using ZPT for my georss feed, but for posterity, there are at least two possible workarounds:

1. Put an xml declaration at the top of your template (this makes the parser to use XML mode). If  your template is only for a snippet and not for a whole document, you don’t want the declaration showing up in the middle of the final document. To avoid that, use METAL macros.  It’s a bit tricky to do this from python, so we have a handy render_tal function in opencore.browser.tal.  And you’ll need to manually encode to your output encoding if you want to use this with response.write().  An example of this that I later reverted can be seen here.

2. You could, if you don’t want to use macros and can’t have the xml declaration in your output, instead subclass Five’s ZopeTwoPageTemplateFile like so to force the TAL parser to use the right mode. (Thi sis also in the example linked above.)

 class XmlPageTemplateFile(ZopeTwoPageTemplateFile):

    """This class exists because the process whereby PageTemplateFile
    decides whether to use HTML or XML mode for parsing and validation
    is completely stupid.
    """

    content_type = 'text/xml'

    def _cook_check(self):
        # In PageTemplateFile, this gets called potentially multiple
        # times (if you're in dev mode), sniffs the file's content,
        # and if it doesn't start with an XML declaration, uses HTML
        # mode.  We don't want that.  So here, we duplicate the same
        # code without the sniffing.
        if self._v_last_read and not DevelopmentMode:
            return
        __traceback_info__ = self.filename
        try:
            mtime = os.path.getmtime(self.filename)
        except OSError:
            mtime = 0
        if self._v_program is not None and mtime == self._v_last_read:
            return
        t = self.content_type  # Take my word for it, dammit!
        f = open(self.filename, "rb")
        if t != "text/xml":
            # For HTML, we really want the file read in text mode:
            f.close()
            f = open(self.filename, 'U')
            text = ''
        text = f.read()
        f.close()
        self.pt_edit(text, t)
        self._cook()
        if self._v_errors:
            logger.error('Error in template: %s' % 'n'.join(self._v_errors))
            return
        self._v_last_read = mtime

For my georss feed, I ended up deciding this was all too much bother and I just went back to using Python string interpolation. There just wasn’t enough to the “template” to justify the pain of getting TAL to work properly under these constraints.