OpenPlans Operations

All,

I spent some of today (likely yesterday unless you’re up this late) talking with Nick, Anil, and other stakeholders/developers about what they’d like covered for our June 2nd-ish deployment. Specificaly, I wanted to know where they thought the risks were — i.e., what changes/new features we need to assess — as well as what they were worried about. Here’s what I found (higher priority first):

1. Everything related to Vacuum. Users should feel like they are logged in no matter what site their on, and their “identity” / user persona should reamin clear. (For instance, long-time StreetsBlog reader signs up with Groups, and now wants all his new comments displayed on StreetsBlog. This is an important use case.)
2. OpenCore Template changes. We’ve got a new look and feel. In general, how does this affect our system?
3. First 10 minutes. We pushed hard on this in earlier testing. This is still very important. Let’s push hard again.
4. Group summary page. This is the single piont of entry for most new projects. Let’s see if it’s solid.
5. StreetsWiki’s debut. StreetsWiki is finally showing her face. Let’s meet her and see how she holds up.
6. Streetsblog, and it’s widgets. They’re big home-grown moving parts. Do they do what we expect?
7. StreetFilms, and it’s widgets. Yes, StreetFilms too.
   

As some of you may know, I’m currently spending three days a week doing geo-related development, and am spending the other two days doing testing related work. I realized I would only get four days (roughly) for testing until we deploy — not enough for the risk areas described above — so I took the liberty of writing charters so I could direct others in the testing effort.

You can find the list of charters here: http://trac.openplans.org/openplans/wiki/JuneSecondDeployment

Since testing is an investigative process, it’s likely that we won’t get all the information we need on the first go. Many of these charters may need to be done multiple times — we’ve done the joining charters many times before — in order to assess the quality of a given feature or product component.

From our previous attempt at exploratory testing (what some may call our charter-based testing), we were, at best, able to complete two charters a day per person. Of the charters given above — not withstanding any new charters we may find in the process — I’d need at least seven days to get through the initial run on my own. Taking into account multiple eyes, innatentional blindness, skill level, relation to the product, etc., I’d need many more days and/or more people.

Though I like to be called the Assurinator, I’d like to stress that a tester cannot assure the quality of the product; we can only provide stakeholders with information. Given that we only have two weeks until we deploy, I’d argue we need to get them this information as soon as possible.

I’ll be sending an email out to stakeholders to let them know of this dependency, but I wanted to list the testing team’s general plan of attack so we can move quickly tomorrow:

1. Tuesday, May 20th: Coordinate/coordinate with whoever will be testing with me, and go over stakeholder risks. Begin testing, giving whoever’s working with me enough work for the Tuesday and Wednesday (I’ll be on geo-stuff Wed.). Report to stakeholders on any findings by the end of the day.
   
2. Thursday. Check-in, do review of our testing effort, re-prioritize/create new charters if needed, and report to stakeholders. Again, I’ll be giving enough charters for Thursday, Friday, and Monday.
   
3. Repeat the above two steps again for the next week, all the time trying to prioritize charters as we gain new information.

Although it feels like we’re under a tight schedule to get this done, I’m actually very excited to do this. I love testing — even when it’s a challenge — and am really starting to feel like the organization is embracing it as well. Thanks to all for letting me give the testing presentation last week (and dwins for setting it up!), and thank you to all who’ll be helping me with testing these next few weeks!

Tim