• OpenPlans Deployment Instuctions

  last modified April 3 by ra


This page provides instructions related to the deployment of development, testing, and production instances of the full stack of the OpenPlans software suite, using the fassembler build tool.

Deployment Checklist


Deployment location and layout

We'll start with an overview of where sites should be deployed, and how the deployments should be organized.

For any given, site, there is a deployment directory.  All files related to this sites operation and deployment should live within the deployment directory.  We keep our deployment directories in /usr/local/topp, and each deployment directory is named with the FQDN of the site that it is serving.  For example, the deployment directory for stage.openplans.org should be /usr/local/topp/stage.openplans.or­g­, (on flow, our development machine).  The deployment directory for www.nycstreets.org  should be /usr/local/topp/www.nycstreets.org, (on theman, our production machine).

Inside each deployment directory there will be a var directory and a builds directory.  The var directory contains data that persists across multiple builds, such as the ZODB database file, and the various log files that the applications generate.  The builds directory should contain a bunch of build directories, each named according to a timestamp of when it was built.  A build directory might have a name of 20080111-1.  Also, in the deployment directory, there should be a symlink to the most recent build directory, called current.  In our example, then, current -> builds/20081101.

Inside a build directory, there will be a number of directories.  Most of these subdirs will be virtualenvs, representing an installation of one of our applications.  Typical directories of this type will include cabochon, opencore, deliverance, tasktracker, wordpress, etc.  Also, every build directory should include a bin directory, an etc directory, and a requirements directory.

The bin directory contains various executables, used for running fassembler, starting specific services, or managing the supervisor daemon. 

The etc directory contains all of the typically changeable configuration information required by the various applications.  This directory and all of its contents are automatically under subversion revision control.

The requirements directory contains a bunch of requirements files, which are used by fassembler to install requirements into the various virtualenv directories.

To recap, a deployment layout should look something like this:

­ 

/usr/local/topp/stage.openplans.org/
 |- builds/
     |- 20080111-1/
         |- bin/
         |- cabochon/
         |- deliverance/
         |- etc/
         |- fassembler/
         |- opencore/
         |- requirements/
         |- scrip­ttranscluder/
         |- supervisor/
         |- tasktracker/
         |- twirlip/
         |- wordpress/
 |- current -> builds/20080111-1
­ |- var/

 

Creating a new deployment 

Most of these directories will be created for you by fassembler, described below.  When bootstrapping a brand new deployment, you should only need to run the following steps: 

 # mkdir /usr/local/topp/site.domain.org
 # cd /usr/local/topp/site.domain.org
 # svn cat https://svn.openplans.org/svn/fassembler/trunk/fassemble­r-boot.py > fassembler-boot.py

To make things even easier, there is a script that will help with this: 

 # /usr/local/topp/build-scripts/newsite.py SITE.DOMAIN.ORG BASE_PORT

In addition to generating the directories, this will create an 'apache.conf' file in the /usr/local/topp/site.domain.org  directory which will contain <VirtualHost> directives for our typical Apache setup.  In most cases this can simply be pasted into the existing httpd.conf file.  You will also need to add a rewrite rule to /etc/squid/squidRewriteRules.py. 

­ Before actually performing the first build for a new site, you will want to set up a new configuration set for that site.  It is possible to skip this step, in which case fassembler will create a default config set for you.  However, it is likely that the default config set will be quite different from what you want.  It is generally better to use 'svn cp' to copy an existing config set from a site that is similar to your new site, to minimize the number of adjustments you will need to make.  For instance, if you're creating 'dev.openplans.org',  but there already exists a config set for 'stage.openplans.org',  then you'd do the following: 

 # svn cp https://svn.openplans.org/svn/config/stage.openplans.org https://svn.openplans.org/svn/config/dev.openplans.org

 Then you'll want to check out the new config set (to anywhere, even your local machine), make any necessary edits, and then commit.  Note that you do NOT have to worry about editing the build directory setting, that will be taken care of when fassembler tries to merge the configuration during the build process, described below.


Creating a new build in an existing deployment

Creating a new build inside an existing deployment consists of the following steps: 

 # cd /usr/local/topp/site.domain.org
 # python fassembler-boot.py builds/YYYYMMDD-1
 # ./builds/YYYYMMDD-1/bin/fassembler var=var etc_svn_subdir=site.domain.org [requirements_svn_repo=https://path/to/requirements] base_port=BASE_PORT db_prefix=site_domain_ -bbuilds/YYYYMMDD-1 fassembler:topp
 # ./builds/YYYYMMDD-1/bin/fassembler var=var etc_svn_subdir=site.domain.org ­[­requirements_svn_repo=https://path/to/requirements] base_port=BASE_PORT db_prefix=site_domain_ -bbuilds/YYYYMMDD-1 all
 # if [ -L current ] ; then rm current; fi
 # ln -s builds/YYYYMMDD-1 current

 In the above fassembler commands, the requirements_svn_repo is the svn URL of the set of requirements files to be used by the various builds.  The default is https://svn.openplans.org/svn/build/requirements/trunk/.   Various openplans releases can be found here: https://svn.openplans.org/svn/build/requirements/releases/.   For nycstreets builds, you'll want to use one of the sets in here: https://svn.openplans.org/svn/build/requirements/nycstreets/.

If you used the newsite.sh script to create your deployment directory, then in your deployment directory there should be a newbuild.sh script.  Running the following will perform the above steps: 

 # /usr/local/topp/site.domain.org/newbuild.sh REQUIREMENTS_SVN_URL

When you create a new build using one of the above methods, you will almost certainly be prompted by fassembler about some differences between the auto-generated configuration and the build configuration that is stored in subversion.   You'll always want to hit the 'd' key and examine these differences carefully.  They are usually one of three categories:

  1. Persistent configuration settings that are unique to this deployment.  These have been committed to svn.  You want to preserve these settings, so you'll generally want to choose 'n' so as to not overwrite the persistent config.

  2. Changes to the build directory.  The build directory changes with each and every build, so this is unfortunately a lot of superfluous noise.  We'll work on minimizing this further going forward.  For now, if the diffs for a file are all of this nature, you should choose 'y' so the config file will be overwritten and the new build directory will be added to the config.

  3. Changes to the shared configuration setup, usually made in the fassembler config skeletons rather than in any single config checkout.  Changes of these natures should be propagated to the saved config, so will usually require a 'y' response.

Note that is is possible that one file could contain diffs of type 1 and of either type 2 or 3.  If this is the case, you must hit 'ctrl-c' to quit the build, hand merge the config changes into the etc/ checkout, commit your changes, and then rerun the fassembler command exactly as it was run before.  Be sure to examine all of the diffs to make sure everything goes smoothly.