• Getting Started

  last modified May 16 by ra


Are you impatient!? You can skip the preliminaries, make sure you have the dependencies, and get right to the ­quick install.


Abstraction (wherein we categorize the following content)


Introduction (wherein we learn the answers to a number of pressing questions)

Where am I?

This page provides instructions for people who are interested in installing their own deployment of the OpenPlans software. OpenPlans is the software which powers openplans.org (<-- warning: recursive link), the very site which you are now visiting.

OpenPlans is free, open source software. It consists of a considerable amount of custom code, built on top of a number of other popular open source technologies. Notable underlying components include Python, Plone, Pylons, Paste, WordPress, and MySQL.

Who is this for?

OpenPlans software, once it's installed, is for anyone who knows how to use a computer. But this page is about installing the OpenPlans software, which requires a bit more savvy. You should at least have some basic familiarity with Linux and/or Unix systems administration. OpenPlans is heavily Python-based, so ideally you'll be somewhat familiar with using and installing Python packages using easy_install.

What is required?

Currently only Unix-like (e.g. Linux, FreeBSD, Mac OS X) environments are supported by the OpenPlans software. While in theory there is nothing that would prevent the software from being made to run in a Windows environment, since the current developers only use Unix environments for development and deployment, all of the existing build tools are a bit Unix-centric. If you are interested in getting OpenPlans to work in a Windows environment, let us know on the dev list.

Additionally, OpenPlans depends on the availability of a number of other tools, including Python (2.4.3 - 2.4.5), Subversion, Apache, PHP, and MySQL. These must be installed and available to the OpenPlans installation tools or the installation will not succeed.

What does the installation process entail?

Assuming all of the prerequisite dependencies are installed and correctly configured, the process of getting the OpenPlans software installed is pretty straightforward. It consists of the following steps:

  1. Install the OpenplansBoot package.
  2. Run the 'new_openplans_site' command to bootstrap a directory structure for managing an OpenPlans deployment.
  3. Run the 'newbuild.sh' script generated by the new_openplans_site command to launch a build of the OpenPlans software.

That's it! Once the build completes, you should have a fully working OpenPlans site, ready to start up and start accepting requests.

Is it really that easy?

Well, yes and no. Those are the steps involved. But the devil is in the details. It may take some twiddling to get the prerequisite dependencies set up correctly, if they're not already installed and in use on your target machine. And the build in step 3 is lengthy; there are a number of things that can go wrong during that process, which may require some troubleshooting. Macs in particular can be a bit finicky. Generally, however, it's not too hard to get things going, and after you've got it working the first time, subsequent builds (whether to create a new site or to upgrade an existing) work with very little trouble.

Great! How do I get started?

Glad you asked! Just keep reading... the rest of this page details the exact steps you'll want to take, outlining some pitfalls along the way. Good luck, and we hope you enjoy playing with OpenPlans!

­

Preparation ­(wherein we learn the prerequisite dependencies)

What you will need

We'll start with a list of all of the pieces that need to be in place before you can begin:

  • Python 2.4: You need to have Python 2.4 available on your path (or specified using the extra_path variable). It must be 2.4.3 or greater. Sorry, Python 2.5 is not yet supported.

  • Easy Install: If your Python 2.4 instance does not already support the easy_install command, you'll want to make sure it's available.
  • MySQL: You need to have a running MySQL server and the administrative root password for that server. We develop on mysql 5..0. Earlier versions are not supported.
  • Apache w/ PHP: You must have a version of the Apache web server, containing a functional PHP installation. We normally use apache 2 and php 5.2 with the "hash" module installed, and support for MySQL.) Also, both the Apache httpd and PHP php-cgi binaries must be available on your PATH (or specified using the extra_path variable). Don't worry, OpenPlans will not interfere with an existing Apache server; instead it generates a custom httpd.conf and uses the existing Apache binary to launch an instance listening on a different port for its own needs.
  • Subversion: Most of the OpenPlans software is retrieved from the internet using Subversion, so 'svn' (with SSL support) must be available on your PATH (or specified using the extra_path variable).
  • A config svn repository: In addition to retrieving software, OpenPlans uses Subversion to manage the configuration files associated with its various services. You will need to specify the URL of a working Subversion repository that OpenPlans can use for this purpose. WARNING: This configuration will contain sensitive system information, including passwords to some of the OpenPlans services. The svn repository you use should NOT be available to untrusted users. If you have a full subversion installed on your deployment machine, it is very simple to use "svnadmin create /path/to/config/repo" to create a repository and then "file:///path/to/config/repo" as your repository URL.
  • Miscellaneous other unix tools: Our build scripts assume you have wget, bash, tar, patch, make. You probably have most or all of these, but notably, Macs do not have wget installed by default. If you discover any other command that the build requires which isn't installed on your system, let us know on the dev list so we can update this page.
  • libxml2
  • libxslt
  • libjpeg (On Gentoo systems, emerge media-libs/jpeg)

Instruction (wherein we cover the installation process in no small detail)

Now that you've got the prerequisite pieces in place and functional, you're ready to get to the actual process of installing OpenPlans, via the following steps:
  1. Install the OpenplansBoot package: This bootstraps the entire installation process. OpenplansBoot is installed with the following command:

    $ easy_install https://svn.openplans.org/svn/OpenplansBoot/trunk
    

    We recommend you use the ­virtualenv package to install OpenplansBoot into a virtual Python environment, rather than into the system Python environment. Regardless of whether you do this, OpenPlans will make heavy use of virtualenv for managing its own services.

    As noted above, Python 2.5 or later is not supported at this time. Many systems use 2.5 as the default Python these days, so you may need to explicitly use easy_install-2.4 to ensure that your build uses the right Python version.
  2. Run the 'new_openplans_site' command: Installing OpenplansBoot should put a 'new_openplans_site' command on your executable path. It should be invoked as such:

    $ new_openplans_site [--use-plone3] site.domain.com base_port
    

    Where site.domain.com is the hostname at which you plan on making your OpenPlans site available (you can choose any descriptive name if you're just trying to test it out locally) and base_port is the numeric port at which the site will be made available. WARNING: OpenPlans actually consists of many interacting HTTP services, all exposed as one integrated site available at the base_port. Each of these services needs its own port, so you should choose a base_port that is followed by a space of available ports. At the time of this writing, OpenPlans uses 8 ports, from base_port through base_port+7, although you may want to leave even more available space to support additional services that OpenPlans may add in the future.

    A significant amount of OpenPlans functionality is in the OpenCore product, built on top of Plone. Currently, openplans.org is using Plone 2.5.5 (more or less... ;-), but if you're interested in living on the bleeding edge, you can use the --use-plone3 flag to specify that you'd like to use the experimental Plone 3.0.6-based version of OpenCore.

    Immediately upon running the new_openplans_site command, you'll be prompted for the URL of your configuration svn repository. Once this is provided, the command should return almost immediately, having created a directory with the name of your specified hostname. This directory will contain a sparse OpenPlans deployment skeleton directory structure, as well as the 'newbuild.sh' script that you will use in the next step.
  3. Run the 'newbuild.sh' script: Okay, now we get to the actual meat of the process. The newbuild.sh script launches a lengthy build process, which will, after a bit of time, generate a fully working OpenPlans installation, ready to be launched and to start accepting requests. You can go get a snack, but you shouldn't wander too far because you will be prompted for a bit more information during the process. Here's how you invoke the command:

    $ cd site.domain.com
    $ ./newbuild.sh requirements_directory
    In the above command, requirements_directory is the name of a subversion directory containing a lengthy specification of all of the underlying dependencies of the OpenPlans software. OpenPlans has a number of different build configurations, this parameter allows us to easily choose between them when we are building the software. The simplest choice for getting started is the following:

    $ ./newbuild.sh openplans/trunk
    If you run newbuild.sh with no arguments, you will see a list of the available requirements directories.

    NOTE: If you used the --use-plone3 flag in step 2, then you must use the following newbuild.sh invocation or your installation will not work:

    $ ./newbuild.sh openplans/branches/plone30
    After executing newbuild.sh, there will be a short delay as initial requirements are downloaded. Eventually you will be prompted for an admin password that will protect your OpenPlans installation. You can enter a password of your choosing (twice), or you can simply hit 'Enter' to autogenerate a random password. In either case, the password will be stored in an admin.txt file that you can refer to later.

    Once the admin password has been entered, the build will continue to churn. The next thing you will be prompted for is the MySQL root password. This is needed because OpenPlans will be creating a number of MySQL databases to support some of its services.

    There's one more issue that may arise; some of the software is downloaded from the Plone subversion repository, which is protected by an unsigned SSL certificate. If you have never connected to this repository, then you will be prompted about the certificate, and given a chance to accept or reject it. If this happens, we recommend you hit 'p' to 'p'ermanently accept the certificate . Or you can prevent this from happening altogether by running the following command before running newbuild.sh:

    $ svn ls https://svn.plone.org/svn/plone
    This will also prompt you to accept the certificate, if you haven't already done so; if you choose 'p' at this point, then it will not come up again while running the build.

    That's it! Assuming that all of the prerequisites are configured and available to the build process as needed, then your build will complete and you will have a fully functional OpenPlans installation. Your build will live within the 'builds' directory, in a subdirectory named after the current date. Let's fire it up!
  4. Start the supervisor daemon: OpenPlans uses supervisor to manage the various services of which it is comprised. To start OpenPlans, then, you use the following commands:

    $ cd builds/YYYYMMDD
    $ ./bin/supervisord
    This starts the supervisor daemon. You can examine the processes that are being managed using:

    $ ./bin/supervisorctl status
    You would also use 'supervisorctl' to stop, start, and restart the services, either individually (by service name, e.g. 'supervisorctl restart opencore') or as a whole (using 'supervisorctl restart all'). If you want to shut down the entire rig, including the supervisor daemon itself, use:

    $ ./bin/supervisorctl shutdown
  5. Use the OpenPlans site: Assuming you haven't stopped or shutdown the services, you should now be able to point a web browser at http://localhost:base_port and immediately start using the OpenPlans site. C'est finis. :-)

Reduction (wherein we list the installation steps w/o all of the extraneous mess)

Those instructions seem more daunting than they actually are. If we remove all of the (hopefully) helpful extra information and reduce it to the commands that you need to enter, it looks like this:

$ easy_install-2.4 https://svn.openplans.org/svn/OpenplansBoot/trunk
$ new_openplans_site site.domain.com  base_port  # (<-- enter config svn repo URL)
$ cd site.domain.com
$ ./newbuild.sh openplans/trunk  # (<-- enter admin pw, mysql root pw, possibly 'p' to accept ssl certificate)
$ cd builds/YYYYMMDD
$ ./bin/supervisord

Or, if you want a Plone 3-based installation:

$ easy_install-2.4 https://svn.openplans.org/svn/OpenplansBoot/trunk
$ new_openplans_site --use-plone3 site.domain.com  base_port  # (<-- enter config svn repo URL)
$ cd site.domain.com
$ ./newbuild.sh openplans/branches/plone30  # (<-- enter admin pw, mysql root pw, maybe 'p' to accept ssl certificate)
$ cd builds/YYYYMMDD
$ ./bin/supervisord

Then point your browser at http://localhost:base_port/. Not so bad, eh?


Graduation (wherein we learn about configurability and tweakage)

Chances are good that, if you're going to manage a few different OpenPlans installations, you're going to use a single configuration repository for all of your configuration sets. It's a bit annoying to have to interactively enter the svn repository URL each time you run the new_openplans_site command. Luckily (you can probably see where this is going...) you don't have to. OpenplansBoot provides a bit of configurability to let you specify some specific build parameters ahead of time, and to persist them so that different installations will share certain attributes. This section gives a brief overview of how you can twiddle some knobs.

Creating the newsite.ini file

When you easy_install the OpenplansBoot package, Python installs an "egg" into your Python environment. This egg is really just a directory, containing the package's Python code, as well as certain metadata associated with the package. It's beyond the scope of this document to go into too much detail, but this egg directory can usually be found in your Python environment's lib/python/site-packages directory. It will have a name similar to "OpenplansBoot-0.1dev_r16550-py2.4.egg" (the version number and revision number may vary, obviously). If we call this directory the egg_root, then you should be able to find a sample configuration file at egg_root/openplansboot/newsite.ini.example. This file will contain some sample configuration settings for new builds. The example file has no effect, however; in order to actually impact your builds, you need to 'cp newsite.ini.example newsite.ini' and then tweak the settings in newsite.ini to your liking. If the newsite.ini file exists in this location, it will be used by the new_openplans_site command when bootstrapping the deployments.

What settings can I change?

You should refer to the newsite.ini.example file for a complete description of what settings are available to you and what effects they have. I'll describe a couple of the more useful ones here to get you started:

  • base_dir: At TOPP, we keep all of our OpenPlans deployments in a single location on our servers. If the base_dir variable is set, then new_openplans_site will create the deployment skeleton in the specified location, instead of in the current working directory.
  • req_user: Similarly, TOPP uses a single user account to manage a single OpenPlans installation. If req_user is set, then both new_openplans_site and the generated newbuild.sh file will check the current user id and will exit with an error if it does not match the specified req_user.
  • extra_path: Sometimes certain tools that the OpenPlans build needs access to live in places that are not on a standard user's execution path. Any specified extra_path value will be appended to the current user's execution path for use during the build process. Multiple extra paths should be separated by a colon (":") as is standard for the PATH shell environment variable.
  • force_ssl: OpenPlans has support for forcing all logins and password changes to be forced through a secure SSL connection rather than the standard unencrypted HTTP. However, getting this to work depends on having a correctly configured SSL server in front of the OpenPlans installation; OpenPlans itself doesn't handle SSL handshaking. If you know that your deployment will have an SSL front-end and you want to activate this feature, set force_ssl to 'True'. NOTE: It is also possible to change this as a per-build setting later, so don't worry if you want to get started without it, you won't be locked in.
  • etc_svn_repo: This is the variable that specifies the configuration repository URL. If this is set, you will not be prompted to enter the URL when you run new_openplans_site.
  • etc_svn_prefix: If many developers are sharing a single configuration repository, it's possible that there will be namespace collisions between their installations. If two people use 'test' as the name of their deployment, then the OpenPlans build will get confused and will try to use the same configuration checkout for the two distinct installations. When you set the etc_svn_prefix variable, the OpenPlans build will add the specified string prefix to all of your configuration checkins, greatly reducing the chance of conflicting with someone else.

It should also be noted that ALL of these settings, and any others that are supported by newsite.ini, can also be entered on the command line. For instance, if you want to place one single OpenPlans deployment in a different location than the rest, and to force it to use SSL when usually you do not, then you could use the following invocation:

$ new_openplans_site site.domain.com base_port base_dir=/path/to/special/place force_ssl=True

Any setting that is specified on the command line will override the same setting specified in the newsite.ini file.

What about the MySQL root password?

The only remaining piece of information that you have to enter interactively for a new deployment, but which is unlikely to change very often, is the MySQL root password. Because this is a particularly sensitive piece of information, we did not add support for it in the newsite.ini file, nor do we allow it to be entered on the command line. If you don't want to enter this every time, then you will want to create a file named .mysql-root-pw in the home directory of the user that is running the newbuild.sh command. This file MUST be chmod 600, i.e. not readable by other users on the system. If the file exists but is not chmod 600, then the OpenPlans build will exit with an error.

Dysfunction (wherein we learn what to do if things go horribly wrong)

Despite all of our careful preparation and instruction, it is, alas, an imperfect world, and you may find that things do not work as smoothly as intended. What should you do to recover, and/or to get assistance if you don't understand what's going wrong?

D'oh

We'll start with the easier case. What if you're running a build using newbuild.sh, and it fails for an obvious reason, like (for instance) that you forgot to start the MySQL server before running the build. You start the server up, and aren't sure where to go next. You could re-run the newbuild.sh script... but that would start a new build in builds/YYYYMMDD-1, instead of completing the earlier build. If you want to try to pick up the original build where it left off, you can run the following commands:

$ cd builds/YYYYMMDD

$ source fassembler/bin/activate

$ fassembler missing

This should restart the build where it left off. When it's complete, use the following commands to reset your Python environment and start up OpenPlans:

$ deactivate

$ ./bin/supervisord

WTF?!

Unfortunately, some problems may not have an immediately obvious solution. In fact, you may not have any idea at all what's going on. In that case, OpenPlans developers may be able to help you past your hurdles. There are two primary mechanisms for getting support from the OpenPlans developers. The first is via the opencore-dev mailing list. This is a very active mailing list that is used by all of the primary OpenPlans developers. It is strongly recommended that anyone who is using the OpenPlans software be subscribed to this list in order to keep up with what's going on in OpenPlans development. It is also a fine place to ask questions. Please make sure you include lots of information about the specific problem you're having, including your underlying platform and any error messages that you might be seeing. You may find some help with how to formulate a helpful question here.

You can also find the OpenPlans developers on IRC, in the #openplans channel of irc.freenode.net. Again, this is a very active channel that is used almost daily by the OpenPlans developers. Most of us are in the U.S. and are on-line during the workday, so please plan accordingly if you're in another part of the world and are trying to reach us.