I’ve been working over the last week on fassembler’s conflict management for configuration files. As I mentioned in an email to the fassembler mailing list, I’ve been working on integrating Gentoo’s wonderful etc-update script. This script works by saving new versions of a config using a special mangled syntax (e.g. cabochon.ini becomes ._cfg0000_cabochon.ini), and then using standard system tools for diffing, merging, and hand-editing conflicts.
When embedding it into fassembler, however, I ran into a two major difficulties. The first was handling configuration on a per-environment basis. A few simple modifications later, and etc-update no longer queries portage, but looks for config relative to the installed script file. The second was stickier, as a number of fassembler projects contains steps not just for building the software, but also for performing data modifications to make it easier to go from zero to working for out of the box installs. In order for these steps to work, the config has to already be in place, or starting necessary services will faill in a spectacular fashion.
This led me to create the concept of phases for fassembler. In the etc-update branch of fassembler, every task has a phase, inherited from the base class, defaulting to ‘build’. A few tasks are overriden at the class level, so that they are automatically placed into the ’setup’ phase. (Running Zopectl scripts is a task like this). Finally, some tasks allow you to specify the phase as part of their init functions. (A good example of this is the Script task. It defaults to the build phase, but if your are running a script that populates a database, for example, you can specify phase=’setup’, and that particular script will be deffered to the setup phase.
The phases are left as a text field to allow more flexibility in the future. Right now, the build phase is triggered, then etc-update is launched, and finally the setup phase is triggered. In the future, it could be possible to add pre and post hooks for every phase, or allow custom phases, but I’ll let that happen if there’s ever a need.
I want to present an example run on my system of opencore 0.12.1. The first text below is what happens when you build using fassembler trunk:
Start
0:00
douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1
------------------------------------- SNIP -----------------------------------------
Prompt 1
+2:42
Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] d
--- /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (old content)
+++ /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (new content)
@@ -6,7 +6,7 @@
email_to = ianb@openplans.org
smtp_server = localhost
error_email_from = paste@cezan.openplans.org
-base_path = /home/douglas/topp/etc/builds/20080807-1
+base_path = /home/douglas/topp/etc/builds/20080813
[server:main]
use = egg:Paste#http
@@ -16,7 +16,7 @@
[app:main]
use = egg:DeliveranceVHoster
data_dir = /home/douglas/topp/etc/var/deliverance/hosts
-custom_error_dir = /home/douglas/topp/etc/builds/20080807-1/deliverance/src/deliverancevhoster/errors/www.openplans.org
+custom_error_dir = /home/douglas/topp/etc/builds/20080813/deliverance/src/deliverancevhoster/errors/www.openplans.org
#debug_headers = true
# To view bodies:
#debug_bodies = true
Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 2
+2:50
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 3
+3:30
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 4
+4:12
Overwrite /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 5
+4:18
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 6
+5:03
Overwrite /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 7
+5:09
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 8
+5:17
Overwrite /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 9
+5:33
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 10
+9:36
Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 11
+9:43
Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] d
--- /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (old content)
+++ /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (new content)
@@ -1,12 +1,14 @@
-
+
+
False
-
+
- OpenPlans
+ OpenCore Site
- greetings@openplans.org
+ greetings@localhost.localdomain
Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] n
------------------------------------- SNIP -----------------------------------------
Prompt 12
+9:51
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 13
+9:59
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 14
+11:31
Overwrite /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 15
+11:43
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
Prompt 16
+13:16
Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini [y/n/d/b/m/?] y
------------------------------------- SNIP -----------------------------------------
FINISHED
+13:37
Installation successful.
douglas@cezan ~/topp/etc $
As you can see, over a period of 13 and a half minutes, you are prompted to make decisions 16 times. Not only do you have to know which changes to accept and reject, but that’s time you have to spend dedicated to that build process. If you need to be doing other things, the build time quickly increases as you was time waiting for each step.
Below, is an example build using the etc-update branch
Start
0:00
douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1
------------------------------------- SNIP -----------------------------------------
Prompt 1
+11:58
Scanning Configuration files...
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini
Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf
The following is the list of files which need updating, each
configuration file is followed by a list of possible replacement files.
1) /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (1)
2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
Please select a file to edit by entering the corresponding number.
(don't use -3, -5, -7 or -9 if you're unsure what to do)
(-1 to exit) (-3 to auto merge all remaining files)
(-5 to auto-merge AND not use 'mv -i')
(-7 to discard all updates)
(-9 to discard all updates AND not use 'rm -i'): 1
File: /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
1) Replace original with update
2) Delete update, keeping original as is
3) Interactively merge original with update
4) Show differences again
Please select from the menu above (-1 to ignore this update): 2
Deleting /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
rm: remove regular file `/home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml'? y
The following is the list of files which need updating, each
configuration file is followed by a list of possible replacement files.
2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
Please select a file to edit by entering the corresponding number.
(don't use -3, -5, -7 or -9 if you're unsure what to do)
(-1 to exit) (-3 to auto merge all remaining files)
(-5 to auto-merge AND not use 'mv -i')
(-7 to discard all updates)
(-9 to discard all updates AND not use 'rm -i'): 2
File: /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
1) Replace original with update
2) Delete update, keeping original as is
3) Interactively merge original with update
4) Show differences again
Please select from the menu above (-1 to ignore this update): 1
Replacing /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py with /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
mv: overwrite `/home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py'? y
Exiting: Nothing left to do; exiting. 
------------------------------------- SNIP -----------------------------------------
FINISHED
+12:53
Installation successful.
douglas@cezan ~/topp/etc $
The build was started. After twelve minutes, the system scans over the config files, determines which ones don’t need user attention, and prompts you for the only two files with any significant difference. Another minute later, and the setup tasks have complted.
When this completed last night, and I realized the amount of effort it would save me, it nearly brought a tear to my eye… 
Now this post is titled a glimpse from the future, and that’s because the branch isn’t mature enough to merge it to trunk. There are still a couple of questions left before I’m ready to enable it for anyone. For example, fassembler keeps track of which projects are completed, and doesn’t rerun them when using the missing keyword. I’ve got to verify that the project isn’t marked complete until all of it’s phases have finished. Also, there isn’t yet a way to trigger a specific phase from the command line, so it’s an all or nothing approach. Finally, when running in non-interactive mode, the system stops after the build phase, which combined with the problem above, there’s no way to repeat the setup phase. Fortunately for me, this allows me to run builds without worrying about starting zeo on the same port, or modifying live data, but not everyone is building in exactly the same way as me.
All of my future builds will be using this branch (which you can acheive by switching fassembler over to the branch https://svn.openplans.org/svn/fassembler/branches/etc-update ). I hope that in the coming week or two, I’ll be able to polish it up enough to merge back to trunk…
Happy Building!
No Comments
RSSNo comments yet.
Leave a comment