OpenPlans Operations

As TOPP matures, we’ve been doing more deliberate QA work, spearheaded largely by Tim Coulter (yay).

But as Rob Miller noted in a recent comment on Ethan’s blog, we’ve been getting behind in QA.

As I see it, this backlog is the product of two factors: the increasingly vague schedule, and the lack of a dedicated QA staff.

To the first point: Nobody in software really likes schedules (”it’s just not ready yet!”), but I do think they help to maintain focus. As long as you don’t end up putting the schedule ahead of the product.

So, what about testing? Spolsky suggests 1 dedicated tester for every 2 or 3 developers. Does that sound high? I don’t know, but can we afford to keep growing our development staff into the dozens with zero dedicated testing staff? No. I’m convinced that’s a false economy and it’s holding us back.

Let’s look at the alternatives. I can only think of these:

• Spend less time on QA.
  It would be nice to make the process faster, and it would be nice to magically turn our manual test runs into regression tests, but as Tim will surely point out, you can’t skip the manual tests. Regression tests serve an entirely different purpose. Skimping on the time-consuming manual testing is a false economy. Less time on QA means lower quality.
  
• Or, Spend less time on design and development.
  In the short term, I think we’ll be forced to keep doing QA with the staff we have - which means time away from our “primary” responsibilities.
  
• Or, Hire testers.
  Doesn’t sound so bad now, does it?

I’ve been working over the last week on fassembler’s conflict management for configuration files.  As I mentioned in an email to the fassembler mailing list, I’ve been working on integrating Gentoo’s wonderful etc-update script.  This script works by saving new versions of a config using a special mangled syntax (e.g. cabochon.ini becomes ._cfg0000_cabochon.ini), and then using standard system tools for diffing, merging, and hand-editing conflicts.

When embedding it into fassembler, however, I ran into a two major difficulties.  The first was handling configuration on a per-environment basis.  A few simple modifications later, and etc-update no longer queries portage, but looks for config relative to the installed script file.  The second was stickier, as a number of fassembler projects contains steps not just for building the software, but also for performing data modifications to make it easier to go from zero to working for out of the box installs.  In order for these steps to work, the config has to already be in place, or starting necessary services will faill in a spectacular fashion.

This led me to create the concept of phases for fassembler.  In the etc-update branch of fassembler, every task has a phase, inherited from the base class, defaulting to ‘build’.  A few tasks are overriden at the class level, so that they are automatically placed into the ’setup’ phase.  (Running Zopectl scripts is a task like this).  Finally, some tasks allow you to specify the phase as part of their init functions.  (A good example of this is the Script task.  It defaults to the build phase, but if your are running a script that populates a database, for example, you can specify phase=’setup’, and that particular script will be deffered to the setup phase.

The phases are left as a text field to allow more flexibility in the future.  Right now, the build phase is triggered, then etc-update is launched, and finally the setup phase is triggered.  In the future, it could be possible to add pre and post hooks for every phase, or allow custom phases, but I’ll let that happen if there’s ever a need.

I want to present an example run on my system of opencore 0.12.1.  The first text below is what happens when you build using fassembler trunk:

­Start
0:00
    douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1

    ------------------------------------- SNIP -----------------------------------------

Prompt 1
+2:42
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] d
    --- /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (old content)
    +++ /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (new content)
    @@ -6,7 +6,7 @@
     email_to = ianb@openplans.org
     smtp_server = localhost
     error_email_from = paste@cezan.openplans.org
    -base_path = /home/douglas/topp/etc/builds/20080807-1
    +base_path = /home/douglas/topp/etc/builds/20080813

     [server:main]
     use = egg:Paste#http
    @@ -16,7 +16,7 @@
     [app:main]
     use = egg:DeliveranceVHoster
     data_dir = /home/douglas/topp/etc/var/deliverance/hosts
    -custom_error_dir = /home/douglas/topp/etc/builds/20080807-1/deliverance/src/deliverancevhoster/errors/www.openplans.org
    +custom_error_dir = /home/douglas/topp/etc/builds/20080813/deliverance/src/deliverancevhoster/errors/www.openplans.org
     #debug_headers = true
     # To view bodies:
     #debug_bodies = true

    Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 2
+2:50
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 3
+3:30
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 4
+4:12
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 5
+4:18
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 6
+5:03
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 7
+5:09
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 8
+5:17
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 9
+5:33
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 10
+9:36
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 11
+9:43
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] d
    --- /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (old content)
    +++ /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (new content)
    @@ -1,12 +1,14 @@
    -
    +
    +

      False

    -
    + 

    - OpenPlans
    + OpenCore Site

    - greetings@openplans.org
    + greetings@localhost.localdomain

    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] n

    ------------------------------------- SNIP -----------------------------------------

Prompt 12
+9:51
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 13
+9:59
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 14
+11:31
    Overwrite /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 15
+11:43
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 16
+13:16
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

FINISHED
+13:37
    Installation successful.
    douglas@cezan ~/topp/etc $ 

  As you can see, over a period of 13 and a half minutes, you are prompted to make decisions 16 times.  Not only do you have to know which changes to accept and reject, but that’s time you have to spend dedicated to that build process.  If you need to be doing other things, the build time quickly increases as you was time waiting for each step.

Below, is an example build using the etc-update branch

Start
0:00
    douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1

    ------------------------------------- SNIP -----------------------------------------

Prompt 1
+11:58
    Scanning Configuration files...
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf
    The following is the list of files which need updating, each
    configuration file is followed by a list of possible replacement files.
    1) /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (1)
    2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
    Please select a file to edit by entering the corresponding number.
                  (don't use -3, -5, -7 or -9 if you're unsure what to do)
                  (-1 to exit) (-3 to auto merge all remaining files)
                               (-5 to auto-merge AND not use 'mv -i')
                               (-7 to discard all updates)
                               (-9 to discard all updates AND not use 'rm -i'): 1

    File: /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
    1) Replace original with update
    2) Delete update, keeping original as is
    3) Interactively merge original with update
    4) Show differences again
    Please select from the menu above (-1 to ignore this update): 2
    Deleting /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
    rm: remove regular file `/home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml'? y

    The following is the list of files which need updating, each
    configuration file is followed by a list of possible replacement files.
    2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
    Please select a file to edit by entering the corresponding number.
                  (don't use -3, -5, -7 or -9 if you're unsure what to do)
                  (-1 to exit) (-3 to auto merge all remaining files)
                               (-5 to auto-merge AND not use 'mv -i')
                               (-7 to discard all updates)
                               (-9 to discard all updates AND not use 'rm -i'): 2

    File: /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
    1) Replace original with update
    2) Delete update, keeping original as is
    3) Interactively merge original with update
    4) Show differences again
    Please select from the menu above (-1 to ignore this update): 1
    Replacing /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py with /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
    mv: overwrite `/home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py'? y

    Exiting: Nothing left to do; exiting.  

    ------------------------------------- SNIP -----------------------------------------

FINISHED
+12:53
    Installation successful.
    douglas@cezan ~/topp/etc $
­

The build was started.  After twelve minutes, the system scans over the config files, determines which ones don’t need user attention, and prompts you for the only two files with any significant difference­.  Another minute later, and the setup tasks have complted.

 When this completed last night, and I realized the amount of effort it would save me, it nearly brought a tear to my eye…

 Now this post is titled a glimpse from the future, and that’s because the branch isn’t mature enough to merge it to trunk.  There are still a couple of questions left before I’m ready to enable it for anyone.  For example, fassembler keeps track of which projects are completed, and doesn’t rerun them when using the missing keyword.  I’ve got to verify that the project isn’t marked complete until all of it’s phases have finished.  Also, there isn’t yet a way to trigger a specific phase from the command line, so it’s an all or nothing approach.  Finally, when running in non-interactive mode, the system stops after the build phase, which combined with the problem above, there’s no way to repeat the setup phase.  Fortunately for me, this allows me to run builds without worrying about starting zeo on the same port, or modifying live data, but not everyone is building in exactly the same way as me.

All of my future builds will be using this branch (which you can acheive by switching fassembler over to the branch https://svn.openplans.org/svn/fassembler/branches/etc-update  ).  I hope that in the coming week or two, I’ll be able to polish it up enough to merge back to trunk…

Happy Building!

While working on today’s micro-deployment, I decided I need a few diagrams to make it clear to myself what was really going.  Since I went to the effort of making them, I thought I’d share them here with everyone.

Below is the deploy structure we’ve been using for quite a while.  There are some more pieces to the puzzle, but for the purpose of this deployment, they’re irrelevant.  Notice that we have multiple complete setups, but they all exist within an analogous space.  Only one set can be running at a time, and squid is always pointing to whichever stack is running.

The current live deployment is a little bit messy, since we have two side by side setups, but with both zope instances pointing to the same instance zeo.  While providing the side by side deployment that I prefer, the intertangling is messy and just not sustainable.

This is the deplyment we’ll have after today.  There will be a separate zeo instance that both zope instances will point to.  Care has to be kept to give zeo access to any code that performs zeo write conflict resolution.  Currently, the only Product we use that does is the QueueCatalog…

Various discussions and emails after this last deployment have made it obvious that I’m not the only person invested in the deployments.  As such, I want to be a bit more open with what my plans are so as not to step on the toes of other people who access and support our stack.

In the last deployment, I built a second site to increase my level of comfort with the deploy.  It gave me a larger safety net, and a way to deploy without needing as much knowledge of the existing site structure.  Going forward, while I want to be able to run parallel deploys, I’m not locked down to the way things are.  In fact, I already have to clean up the mess of one site connecting to the zeo from a different site.  I’d like to fix this by performing a micro-deploy on Thursday that just moves zeo out of the build so that we can build sites independantly, and as Rob pointed out, run zopes in parallel for higher availablity and load balancing.

 Since it’s so small, and we know have builds handling it, I’d also like to change the Maildrop config to run out of supervisor.  The upside is not worrying about touching the system for a build, and the downside is that we have to make sure to either migrate the spool, or (ideally) leave any existing maildrop running until it flushes its queue.

I’ve just finished a new deploy of the OpenPlans stack to OpenPlans.org.  What makes today’s deploy special is that I decided to build the new site in parallel today, so that we could switch sites without any downtime.  It’s a first step towards breaking zeo out of the build, as I mentioned before, and it means a big drop in the stress level associated with a new release.

There was one small hitch, where I had to switch back to the old site, because the process up till now assumes a shared var directory.  Since deliverance theming comes from cached files in the var directory, I didn’t notice the locally modified changes, and ended up with broken blogs.  No problem, however, since the previous site was entirely untouched, a three second switch back to the old site allowed me the freedom to diagnose and correct the problem without worries.

Today’s release is mostly bug fixes with some features that are now in code, but not exposed to the user.  It also didn’t require any data migrations.  Both of these will complicate the deployments, but I’m happy to be improving the user experience strictly from an operations point of view

As Ethan pointed out in his email to the opencore-users mailing list, I’m taking on the role of OpenPlans Deployment Manager.  (I capitalize because I imagine that someday, this will not be me )

 I wanted to share here just a little bit of what I intend to bring to the role, and where I’m going in the short- to mid-term.  My first deployment is scheduled for Monday of next week, and I’ve put up a mind map of the changes that will be going into the release at OpenPlans Deployments.  With each release, I’ll be generating a similar map, with the hope that Paul will be able to turn this into a changelog valuable to the community at large of opencore developers, users, and testers.

Until such time, if you’ve got any questions as to what’s on OpenPlans, come to me.  If I don’t know, I’ll find out for you.

 While this first release will be following the legacy build structure, in the coming releases, I’ll be pulling zeo out of our site build.  For versions that don’t require data migrations (like the current), we’ll be able to build a parallel stack on a different set of ports, point it to the same database, smoke check it, and then just tell apache and/or squid to gracefully pass new connections over to the new instance.  That’s my dream… zero-downtime updates…

I’d also like to have builds that are relocatable, so that they can be built in one place, and installed elsewhere.  Doing the build diffs has shown me what a long way we have to go here.  (There are lots of scripts that reference a particular python, symlinks with full paths, etc…)  As it it today, we can’t backup a deployment, and I’m not happy with that.  I plan to write a simple script that will spider a build, and cleanup paths.  That’s a hack, so it’s only a short-term solution, but it should make it easier to move towards a long term solution (as the script will contain the information about where we have problems).

Anyway, these are just my initial plans, if you have any ideas, be sure to let me know…

Last weekend the openplans trac was upgraded to version 0.11rc1.  This gives us access to several new features in trac:

 * ticket workflow:  I haven’t looked into this or how we can take advantage of this, but trac 0.11 provides real ticket workflow that is customizable;  so if we need more advanced ticket workflow than new -> assigned -> closed -> reopened -> closed, we can now have it

* genshi:  probably more of a feature for developers than anything user-facing, but trac now uses genshi instead of clearsilver.  Personally, genshi is my favorite templating system (that I’ve seen + used so far), so this is exciting to me.  Also, with the development version of genshi (which I installed), genshi’s transformer can be used to customize templates as a stream instead of simply copying the template and now having an out-of-sync copy with the master.

* access to plugins that require genshi and/or 0.11

* a number of other improvements;  see http://trac.edgewall.org/wiki/TracDev/ReleaseNotes/0.11

I also installed a number of plugins, including some written by TOPPers.  I put these in a requirements file for easy poachability:

https://svn.openplans.org/svn/TracPlugins/plugins.txt

In addition to these tangible benefits, this also puts us on a good track <pun intended=”true”/> to stay current with trac development and reap the benefits of the new architecture and plugins.  As always, trac is a work in progress, so any feedback on what is good/bad/ugly/pretty is appreciated.  I have my own way that trac would “ideally” work, and I’m sure many of you do as well — the challenge is making trac work well for all of our individual use cases, which is probably more a matter of agreeing on how this would be than implementing.

All,

I spent some of today (likely yesterday unless you’re up this late) talking with Nick, Anil, and other stakeholders/developers about what they’d like covered for our June 2nd-ish deployment. Specificaly, I wanted to know where they thought the risks were — i.e., what changes/new features we need to assess — as well as what they were worried about. Here’s what I found (higher priority first):

1. Everything related to Vacuum. Users should feel like they are logged in no matter what site their on, and their “identity” / user persona should reamin clear. (For instance, long-time StreetsBlog reader signs up with Groups, and now wants all his new comments displayed on StreetsBlog. This is an important use case.)
2. OpenCore Template changes. We’ve got a new look and feel. In general, how does this affect our system?
3. First 10 minutes. We pushed hard on this in earlier testing. This is still very important. Let’s push hard again.
4. Group summary page. This is the single piont of entry for most new projects. Let’s see if it’s solid.
5. StreetsWiki’s debut. StreetsWiki is finally showing her face. Let’s meet her and see how she holds up.
6. Streetsblog, and it’s widgets. They’re big home-grown moving parts. Do they do what we expect?
7. StreetFilms, and it’s widgets. Yes, StreetFilms too.
   

As some of you may know, I’m currently spending three days a week doing geo-related development, and am spending the other two days doing testing related work. I realized I would only get four days (roughly) for testing until we deploy — not enough for the risk areas described above — so I took the liberty of writing charters so I could direct others in the testing effort.

You can find the list of charters here: http://trac.openplans.org/openplans/wiki/JuneSecondDeployment

Since testing is an investigative process, it’s likely that we won’t get all the information we need on the first go. Many of these charters may need to be done multiple times — we’ve done the joining charters many times before — in order to assess the quality of a given feature or product component.

From our previous attempt at exploratory testing (what some may call our charter-based testing), we were, at best, able to complete two charters a day per person. Of the charters given above — not withstanding any new charters we may find in the process — I’d need at least seven days to get through the initial run on my own. Taking into account multiple eyes, innatentional blindness, skill level, relation to the product, etc., I’d need many more days and/or more people.

Though I like to be called the Assurinator, I’d like to stress that a tester cannot assure the quality of the product; we can only provide stakeholders with information. Given that we only have two weeks until we deploy, I’d argue we need to get them this information as soon as possible.

I’ll be sending an email out to stakeholders to let them know of this dependency, but I wanted to list the testing team’s general plan of attack so we can move quickly tomorrow:

1. Tuesday, May 20th: Coordinate/coordinate with whoever will be testing with me, and go over stakeholder risks. Begin testing, giving whoever’s working with me enough work for the Tuesday and Wednesday (I’ll be on geo-stuff Wed.). Report to stakeholders on any findings by the end of the day.
   
2. Thursday. Check-in, do review of our testing effort, re-prioritize/create new charters if needed, and report to stakeholders. Again, I’ll be giving enough charters for Thursday, Friday, and Monday.
   
3. Repeat the above two steps again for the next week, all the time trying to prioritize charters as we gain new information.

Although it feels like we’re under a tight schedule to get this done, I’m actually very excited to do this. I love testing — even when it’s a challenge — and am really starting to feel like the organization is embracing it as well. Thanks to all for letting me give the testing presentation last week (and dwins for setting it up!), and thank you to all who’ll be helping me with testing these next few weeks!

Tim

So I’ve been looking at software solutions for the TOPP-talks scheduling ‘problem.’  Really, I’d like something that lets people register as “I’d like to give a talk and can be ready by x date” but it sounds like people think more in terms of “I’d like to give a talk and can be ready with x days notice.”  The wiki is probably a decent thing for this, especially with the minimal level of talk throughput I’m dealing with these days.  Still, my inner geek is screaming “you could get software to do this!”

 Ideally what I’d like is to have people volunteer, list their timing constraints, add in some further time constraints (ie OpenCore meeting and release schedules, other planned TOPP/Lime events) and push a button to see a list of who could talk when.  Then I’d just check off which people I wanted to schedule and their talks would automagically move to some ’scheduled’ queue (which would autoconvert to a ‘past talks’ list when the Big Day arrives) and get that talk out of my way when dealing with schedules in the future.  (For bigger stuff TOPP Talks or its peers might become later, it could be useful to handle event signups and possibly even tourney bracket creation, which I feel are in a similar vein.)

My hunch is that this sort of thing would be useful for other people trying to wrangle volunteers (hence the post here.)  In my web wanderings I found this: http://civicrm.org/ which isn’t really on the mark but still seems worth a look.

Hey all, just saw an interesting blog post on API design and encouraging 3rd-party participation. Thought I’d share. Aaron Seigo is one of the big minds behind KDE and KDE 4’s new “workspace” which consists of a really flexible widgeting system called plasma.

http://aseigo.blogspot.com/2008/04/on-scripting.html