I was looking over this (remember the days when fassembler-discussion didn't
even exist yet?) and was kind of blown away by how much our "ideal console
session" has become a reality. fassembler totally rocks.
---------- Forwarded message ----------
From: Ian Bicking <ianb@...>
Date: Wed, Nov 14, 2007 at 1:29 PM
Subject: Re: [OpenCore Dev] How should our workflow work?
To: opencore-dev@...


Joshua Bronson wrote:
> Hey Ian,
>
> First of all, thanks for asking! It's not often someone says "give me
> your fantasy workflow and I will build it!" I can't wait for fassembler.
>
> As for my two cents, the parts of building our full stack that I've
> found to be the most trouble are:
>
> 1. Having to manually configure settings (especially ports) in n
> different places for each of the n apps, which you've already addressed
> <
http://www.openplans.org/projects/opencore/lists/opencore-dev/archive/2007/10/1193867403904
>:
>
>     /etc/main.ini
>       I'm thinking of this as a location for the global settings for all
>     the projects.  One in particular is a base port, where all the other
>     ports products use are based on that (e.g., deliverance at base_port,
>     Zope at base_port+1, TaskTracker at base_port+2, etc).  Builds could
>     read from this, but I'd prefer programs read it at runtime; but I'm
not
>     sure how that will work.  Too bad support for HTTP over named sockets
>     isn't widely supported.
>
>
> Just a few specific notes off the top of my head on what settings I've
> had to worry about for each app (plus or minus some underscores, less
> obvious stuff first):
>
> * WordPress needs zope_uri, zope_admin_pw (as of today to query zope for
> projects' security policies), topp_secret_filename, database,
> database_username, database_password, and database_host

zope_uri and zope_admin_pw should come from the global configuration.
We'll need to write or find an ini parser for PHP, but that should be
easily enough.  Oh, and a quick search shows that PHP actually has a
builtin, parse_ini_file(filename, [parse_sections]).  So we just have to
do something like this in wp-config.php:

  define('DEPLOY_ETC_DIR', '{{env.base_path}}/etc');
  $conf = parse_ini_file(DEPLOY_ETC_DIR . 'build.ini', true);
  define('ZOPE_URI', $conf['uris']['zope_uri']);

Ideally wp-config.php would take all its (interesting) settings from a
config file in etc/wordpress/wordpress.ini, like the database connection
parameters.

> * TaskTracker needs zope_uri, zope_admin_pw, topp_secret_filename,
> database, and database_user. The database_user's password is currently
> specified on the command line at build time. Note the inconsistency with
> WordPress. Also note that both TaskTracker and WordPress define the
> variable "database" in their [globals], which makes a full-stack build
> currently impossible. We should use better names and better namespacing
> in the future.

The current fassembler:tasktracker build doesn't require any variables.
 But it doesn't actually work; getting it to work means making TT read
the global configuration for zope_uri, zope_admin_pw, and
topp_secret_filename.

You can override settings like '[tasktracker]database=tasktracker', to
specify that you only want to set the option database for tasktracker.
Currently if you just do database=tasktracker during a build that does
both wordpress and tasktracker (like "fassembler fassember:wordpress
fassembler:tasktracker"), they'll share the same database (which isn't
what you want).  Maybe that shouldn't be allowed, though I'm not really
sure how to handle that.  Maybe the default should be based on a
database name prefix instead, so you'd do "fassembler ...
db_prefix=test_"), and then all the databases would be test_wordpress,
test_tasktracker, etc.  That's probably the right way, and an easy
change.  I've made this change in trunk now.

> * Deliverance needs zope_uri, tasktracker_uri, blog_uri, and twirlip_uri.

I plan to rewrite openplans_hooks.py to work entirely off the global
configuration, so that adding a new app will be fairly trivial.

openplans_hooks.py will also find a more proper place in svn, though
where exactly I'm not sure.

> * Zope needs blog_uri, tasktracker_uri, zope_port, zeo_port, zope_user,
> and zope_password. Note that zope_user and zope_password together
> provide the same information that zope_admin_pw above does (or rather
> the file that zope_admin_pw points to). fassembler can probably clean up
> redundancies like this.

zope_user and zope_password will come from the file that contains the
admin username/password.  This is autogenerated when the base layout is
created.  The first time you create a layout you must provide an admin
password.  Defaulting to "admin" seems weird to me.  I could generate a
random password; I suppose since it's easy to read the admin.txt file to
get the password it's not that important that you choose a password
explicitly.

So generally, no configuration should be required to build anything,
except the base_port when first building an environment.  If MySQL has a
root password, that may also be required for the build (to setup the
other MySQL databases and permissions).

fassembler:tasktracker is already doing the full MySQL setup, including
database creating and setting up the permissions.  It's a reusable
command, so WordPress will just use the same code for this.

> * I'm not sure what the story is with Cabochon or Twirlip, but as they
> get integrated into the stack this iteration, we will want to keep them
> in mind.
>
>
> Having the main.ini you mention above should ease this pain, along with
> the automation you propose, e.g. providing a start port and then
> following a scheme of incrementing by one. I generally really like the
> idea of automation and following simple schemes. We should do this for
> our apps that use sql too: automatically create their database, database
> user, and database password according to some scheme such as
> {site}-{version}-{app} ( e.g. "openplans-0.9.7.7-wordpress",
> "nycstreets-1.0.1-tasktracker"). The sql management is totally something
> we could automate with more of these conventions. A fantasy console
session:
>
> # i've just used fassembler to build a full stack for development
> # on the next release of openplans and i've somehow gone and
> # messed up wordpress's database
> $ fassembler reset-sql fassembler:wordpress
> # fassembler takes care of connecting to sql as the right user
> # and dropping and recreating the appropriate table

I think this is something WordPress should provide.  That is, the
WordPress build should write a bin/reset-wordpress script, and that
script should know what is required to do that reset.  For TaskTracker
this is something like "paster setup-app etc/tasktracker/tasktracker.ini
reset=true" (I can't remember the name of the setting exactly), so
that's already a case where it's mostly self-provided.  But I agree that
providing this functionality for all products would be useful.

> Maybe
> $ fassembler reset-zope
> could do the equivalent in zope.
>
>
> The other major build pain has been configuring deliverance by hand.
> First off, having a requirements=[production|development] will be a win,
> since in production it sits behind apache and only accepts connections
> from localhost (meaning we'll have force_port = 80 and host = 127.0.0.1
> <http://127.0.0.1> by default), whereas in development we won't have
> force_port set, host can be 0.0.0.0 <http://0.0.0.0>, we can turn on
> debugging by default, etc.

Mmm... yeah.  I was thinking about production/development simply being a
question of what versions of packages are installed.  And, actually,
there'd be three settings with Rob's recent diagram (the link for which
I cannot find) -- development, release candidate, and production.  And
possibly custom development rigs, when you are working on a branch of a
particular piece of the system.  Though that might just be ad hoc -- you
build a development rig, then do an svn cp/switch on the code you want
to edit.  All "our" code is installed from svn to make this easier, and
when you rebuilt something it'll be careful about svn repositories (it
will note when a svn repository is not the expected repository, and ask
you what to do).

We can definitely put in logic for stuff like force_port.  I think it
will be a separate setting.  I guess we want to distinguish between a
full production stack (with Apache and Squid) and a somewhat simpler
development stack.  But if we do load testing on any version of software
it needs Apache and Squid to be realistic (though without Apache and
Squid it could still be interesting).  So I don't think it's directly
tied to the versions of packages.

> Second, it'd be nice if we could pass fassembler a theme flag for either
> openplans or nycstreets. This would automatically put the right
> theme.html in the static directory of each host directory deliverance
> creates in .../var/lib/deliverance/hosts. With deliverance's already
> nice automation of host setup (e.g. master_site_domain = auto,
> openplans_hooks.rule_data), and with default_theme_uri set to
> "/theme.html", this should take care of the pain of having to ever go
> into var/lib/deliverance/hosts.

Good idea.  This would probably be a setting in the deliverance build.
You can pass settings in on the command-line for any build.  You can
have build configuration files too, but while I coded for it I haven't
ever used a configuration file, and I'm not sure they'll really be
necessary.  I'm trying to build smart defaults in whenever possible, so
I think only a handful of overrides will be used often.

> fassembler could even use this theme flag to check whether to install
> the sputnik package in the opencore workingenv and put the sputnik slugs
> into zope/etc/package-includes. This way we could build a ready-to-run
> full openplans.org <http://openplans.org> or nycstreets.org
> <http://nycstreets.org> production or development stack with a single
> command!

Sputnik should just be a branch, until everything in Sputnik becomes
configuration flags for our system.  Sputnik and nycstreets.org should
only be separate code until we figure out how to merge everything from
it into the trunk.  So maybe it would be like development/rc/production,
where we have a requirements file that selects the Sputnik branch.

I haven't figured out where the requirements files should go exactly.
In svn?  In the packages themselves?  We could have an opencore
requirement file like:

-f http://dist.repoze.org/simple/PIL/
-f <http://dist.repoze.org/simple/PIL/-f> https://svn.openplans.org/eggs/
PIL
Zope==2.98_final
-e ./#egg=opencore
-e https://svn.openplans.org/svn/topp.featurelets/trunk
-e <https://svn.openplans.org/svn/topp.featurelets/trunk-e>
https://svn.openplans.org/svn/topp.utils/trunk

Where "-e ./" means "this repository right here".  Then when that code
is branched, the requirements file is branched along with it.

This doesn't work for the coordinated branching of more than one
product.  E.g., if you made a branch of featurelets and opencore, to be
used together.  Then you'd have to create a new requirements file or
edit the one in your branch to point at the featurelets branch.  Or you
could just do an svn switch and handle it manually, but there wouldn't
be any automatic way for someone else to get the same build.

--
Ian Bicking : ianb@... : Team TOPP Unite! Form of: Willow!


--
Archive:
http://www.openplans.org/projects/opencore/lists/opencore-dev/archive/2007/11/1195061371436
To unsubscribe send an email with subject unsubscribe to
opencore-dev@....  Please contact
opencore-dev-manager@... for questions.

Attachments

text.html (text/html) 14.8 kB