This past week we talked a lot about our release/deployment process, since things have gotten significantly more complicated recently.  We now have several packages that are evolving (opencore & WP, in addition to TT and Listen which have been relatively stable for a while now) and two separate installations to maintain.  It is becoming obvious that we need to become a little bit more formal about how we plan for releases and deploy.  The following sections outline some plans that came out of a lot of discussions last week:

­

Versions for OpenPlans

One issue is keeping track of our progress on OpenPlans, the combined application/website.  Up until now, we've been using the opencore version number to keep track of our overall progress, but this is becoming increasingly less appropriate.  Really, we need a master version number for OpenPlans in order to:

• Track our progress and be able to archive historical milestones
  
• Plan for the future, using a stable set of release milestones (like many other opensource projects do)
• Allow component version numbers to increment independently of the larger project, for developer reasons (adding a new dependency, etc.)

A major question this raises is: what version is OpenPlans at?  There have been suggestions that we're at 1.0 (we have been at this for a while...), but I would argue that we don't want to spoil the fun of a 1.0 release this way.  The diagram above indicates a preference for something like 0.8, to give us some distance from OpenCore's 0.9.7, and to allow for a 1.0 release in the coming months.  We haven't made any decisions  about this yet, but should do so soon.

Once we decide what version OpenPlans is at, we can replace the "version" field in trac with OpenPlans versions, and use trac milestones with these numbers to plan ahead.

In order to make this work properly, we'll need to start creating proper versions for all our components, not just opencore.  Then, in the future, fassembler will just pull opencore 0.9.7, tasktracker 0.5, wp-in-openplans 0.2, etc. in order to build OpenPlans 0.8.5 (for example).

Release schedule

In order to minimize confusion and facilitate better planning, we're also going to adopt a more structured deployment schedule.  The diagram below outlines a process whereby we'll deploy every week, bumping changes from stage to live and from trunk to stage (on release branches).  This way, there should be no question, at any given time, where changes should be committed to, and when those changes will see the light of day.

The following calendar shows a typical release schedule, over the course of a few weeks:

	SUN	MON	TUE	WED	THU	FRI	SAT
w1				deploy 0.6 to live.nycstreets.	deploy 0.6.1 to stage.nycstreets and stage.openplans QA 0.6.1 ->
w2		­deploy 0.6 to live.openplans		deploy 0.6.1 to live.nycstreets	deploy 0.6.2 to stage.nycstreets and stage.openplans  QA 0.6.2 ->
w3		deploy 0.6.1 to live.openplans		deploy 0.6.2 to live.nycstreets	deploy 0.6.3 to stage.nycstreets and stage.openplans  QA 0.6.3 ->

Note that this schedule shows us deploying to openplans after we deploy to NYCstreets.  I think doing staggered deployments like this makes a lot of sense, so that we can focus on only one thing on deployment day, and minimize confusion.  Which site should go live first is a question for debate, I suppose, as the second deployment would (in theory) be more stable, through bugfixes or lessons learned through migrations.

Exceptions

­Without a doubt, there will be exceptions to these rules, and we should be ready to handle them gracefully. 

For example, this week, the folks at TA wanted to send out email invitations to all the people that signed up or working groups at the UWS event.  We had previously created projects on NYCstreets for these groups, but these projects didn't have any members yet.

The TA folks were a little nervous about the 2-step process of joining a project and then joining that project's mailing list (thankfully, the third step of joining the NYCstreets site was consolidated in OpenCore 0.9.7.3), and we mentioned that we had been working on a workflow improvement whereby project members would automatically be joined to the project's mailing list (and removed if they left the project).  The TA folks agreed that having this feature in place would greatly simplify the process and increase the number of new members who actually subscribed to the lists.

So, here was a case where by deploying a feature outside of the normal release schedule would greatly benefit our team. So, instead of waiting for the next scheduled deployment, we finished the feature, merged into a new release branch (trunk was already polluted with unfinished changes), tested, and deployed.  While this broke our standard deployment protocol, it made our clients/teammates at TA really happy, and pushed out a useful new feature right when it was needed.

This also applies to critical bugfixes to the live site, which should always be fixed and deployed immediately, rather than waiting for the next formal release. 

There will often be times when we have to push certain features & fixes live outside of the normal release schedule, and we should be ready to do that when necessary.   It's important to remember that we're primarily producing a website -- our website visitors (as opposed to our software users) don't have to do anything when we update the site, so there's almost no cost to rolling out fixes.  It's true that we're also producing software that people will download & install, but at the moment, they're not our top priority.
 

Hostnames for deployment

Going hand-in-hand with a tighter release schedule is a more regular use of our various testing & live domains.  In order to accommodate our working, testing, and deployment needs, we've sketched out the following plan for deployment domains:

• www.openplans, www.nycstreets
  the live sites; obviously
• stage.openplans, stage.nycstreets
  QA for the following week's deployment.  Any components that have relevant changes would be running off of release branches.  These branches will accept bugfixes only, not new features.
  
• backstage.openplans, backstage.nycstreets
  Deployment showing work that's currently in trunk (scheduled for 2 weeks ahead's deployment.  The purpose of this is so that folks w/o an up-to-date local install (Mouna, Jackie, our partners, etc.) can see the latest stuff.  It is to be expected that this will be broken sometimes.  It would be the release manager's duty to SVN UP these deployments daily.
  
• leapfrog.openplans, leapfrog.nycstreets
  Proposed new deployment used for pushing stuff to live ahead of schedule.   These sites would always be running exactly the same software (and ideally with similar data) as the live site.  Ahead-of-schedule fixes would be tested here before deploying to www.   Changes made here would need to be merged back to trunk and any open release branches.
  

• dev.openplans, dev.nycstreets
  Flexible space for devs to use for various purposes.
  

Release manager

Something else that's been brought up recently is the need for us to formalize the role of the Release Manager somewhat.  This is a really important job, and it's important to have one person with their eye really on the ball.

Ethan did a great job of this with NUI and subsequent deployments, as did Robert during the sputnik releases.  I see this job as somewhat of a rotating role, where different people might take it on depending on the circumstances (I believe this is how it has been discussed in the past).

It seems like the release manager would have the following responsibilities:

• Knowing what features are in / out of an upcoming release, and directing traffic appropriately
• Ensuring that people are diligent about sending in notes of any necessary migrations or deployment notes
  
• Deploying finalized release to live site
• Merging changes from release branches back to trunk
  
• Cutting new release branches for QA
• Deploying ahead-of-schedule releases to the leapfrog sites

There are a lot of specifics about this role that haven't been fleshed out, but I think this is a pretty good start.  During the iteration meeting, we talked about some possible options for communicating deployment details (posting them to the "deployment notes" wiki page, sending an email to the dev list with a standardized subject line (like DEPLOYMENT), having the RM blog about upcoming deployments and people fill the comments section with notes, etc.), but we didn't really come to any decisions regarding that process.

 

Comments? 

Should we include formal QA/Design team review in the weekly breakdown?

Also is there a place where we could find notes about what is new/changed/fixed in each version so when/if we review, wecan focus on making sure these specific components are tested? This could also be exposed to our users so they see what's new and how great we are.