OpenPlans Operations

While working on today’s micro-deployment, I decided I need a few diagrams to make it clear to myself what was really going.  Since I went to the effort of making them, I thought I’d share them here with everyone.

Below is the deploy structure we’ve been using for quite a while.  There are some more pieces to the puzzle, but for the purpose of this deployment, they’re irrelevant.  Notice that we have multiple complete setups, but they all exist within an analogous space.  Only one set can be running at a time, and squid is always pointing to whichever stack is running.

The current live deployment is a little bit messy, since we have two side by side setups, but with both zope instances pointing to the same instance zeo.  While providing the side by side deployment that I prefer, the intertangling is messy and just not sustainable.

This is the deplyment we’ll have after today.  There will be a separate zeo instance that both zope instances will point to.  Care has to be kept to give zeo access to any code that performs zeo write conflict resolution.  Currently, the only Product we use that does is the QueueCatalog…

Various discussions and emails after this last deployment have made it obvious that I’m not the only person invested in the deployments.  As such, I want to be a bit more open with what my plans are so as not to step on the toes of other people who access and support our stack.

In the last deployment, I built a second site to increase my level of comfort with the deploy.  It gave me a larger safety net, and a way to deploy without needing as much knowledge of the existing site structure.  Going forward, while I want to be able to run parallel deploys, I’m not locked down to the way things are.  In fact, I already have to clean up the mess of one site connecting to the zeo from a different site.  I’d like to fix this by performing a micro-deploy on Thursday that just moves zeo out of the build so that we can build sites independantly, and as Rob pointed out, run zopes in parallel for higher availablity and load balancing.

 Since it’s so small, and we know have builds handling it, I’d also like to change the Maildrop config to run out of supervisor.  The upside is not worrying about touching the system for a build, and the downside is that we have to make sure to either migrate the spool, or (ideally) leave any existing maildrop running until it flushes its queue.

I’ve just finished a new deploy of the OpenPlans stack to OpenPlans.org.  What makes today’s deploy special is that I decided to build the new site in parallel today, so that we could switch sites without any downtime.  It’s a first step towards breaking zeo out of the build, as I mentioned before, and it means a big drop in the stress level associated with a new release.

There was one small hitch, where I had to switch back to the old site, because the process up till now assumes a shared var directory.  Since deliverance theming comes from cached files in the var directory, I didn’t notice the locally modified changes, and ended up with broken blogs.  No problem, however, since the previous site was entirely untouched, a three second switch back to the old site allowed me the freedom to diagnose and correct the problem without worries.

Today’s release is mostly bug fixes with some features that are now in code, but not exposed to the user.  It also didn’t require any data migrations.  Both of these will complicate the deployments, but I’m happy to be improving the user experience strictly from an operations point of view

As Ethan pointed out in his email to the opencore-users mailing list, I’m taking on the role of OpenPlans Deployment Manager.  (I capitalize because I imagine that someday, this will not be me )

 I wanted to share here just a little bit of what I intend to bring to the role, and where I’m going in the short- to mid-term.  My first deployment is scheduled for Monday of next week, and I’ve put up a mind map of the changes that will be going into the release at OpenPlans Deployments.  With each release, I’ll be generating a similar map, with the hope that Paul will be able to turn this into a changelog valuable to the community at large of opencore developers, users, and testers.

Until such time, if you’ve got any questions as to what’s on OpenPlans, come to me.  If I don’t know, I’ll find out for you.

 While this first release will be following the legacy build structure, in the coming releases, I’ll be pulling zeo out of our site build.  For versions that don’t require data migrations (like the current), we’ll be able to build a parallel stack on a different set of ports, point it to the same database, smoke check it, and then just tell apache and/or squid to gracefully pass new connections over to the new instance.  That’s my dream… zero-downtime updates…

I’d also like to have builds that are relocatable, so that they can be built in one place, and installed elsewhere.  Doing the build diffs has shown me what a long way we have to go here.  (There are lots of scripts that reference a particular python, symlinks with full paths, etc…)  As it it today, we can’t backup a deployment, and I’m not happy with that.  I plan to write a simple script that will spider a build, and cleanup paths.  That’s a hack, so it’s only a short-term solution, but it should make it easier to move towards a long term solution (as the script will contain the information about where we have problems).

Anyway, these are just my initial plans, if you have any ideas, be sure to let me know…