Purpose 

In TOPP's recent history -- from the time I've been here, but also before then -- TOPP has released versions of OpenPlans that resulted in differing user reactions. Sometimes, these reactions were positive, expressing the delight of users; other times, however, these reactions were not so positive. Woonerf, our next release coming this March, is more than just OpenPlans: it's also StreetsBlog, StreetsWiki, StreetFilms, and the culmination of Geoserver plus Vespucci. This release, more than any other in TOPP's history, brings together everything TOPP has created since its inception. In order to ensure that Woonerf is as successful and as positive as we all expect, I've been asked to lead a testing effort that is in addition to the current testing efforts within TOPP. The purpose of this proposal is to communicate the scope of this new effort, as well as describe its process and how it fits within the organization. 

Constraints/Focus

Alright, let's be honest. Although I'd love to be able to say the words "Yes, OpenPlans is fully tested," or "Yes, I have enough time to completely test that," the truth is there is never enough time to test everything in every possible way. There is no way that I, or a team of people, will be able to test every new and old part of Woonerf before our release in March. There just isn't enough time. Instead of mechanically trying to test every piece of Woonerf with every possible input I could fathom, in every order, and in every configuration, the testing effort will instead focus on risks, and the things that seem most important to stakeholders.

Who are the stakeholders? From a purely organizational perspective, these would be the "head honchos", the "big wigs", the people in charge. Although TOPP is a relatively flat organization, this means Mark, Jackie, Nick, and Cholmes. Are these the only stakeholders? No, of course not. A stakeholder in the sense that I mean it is anyone who has some say in our software process. This includes users -- both the good ones, the bad ones, and the malicious -- as well as every developer and every employee at TOPP.

If you take my statement above -- the one that says "the testing group will focus on the risks that are most important to stakeholders" -- and apply it to my definition of "stakeholder," you'll realize we have a lot of people to please. Every stakeholder is affected by our software in different ways, and every stakeholder wants a certain part of the system to be the most solid. Because, as stated above, the testing group can't test everything, it therefore can't please everybody; instead, it must prioritize its testing in order to achieve the most benefit for those involved.

Implementation 

In the grand scheme of things, the amount time given to the testing group for the Woonerf release isn't that long. Don't get me wrong -- it's a great amount of time compared to the time given to previous releases -- but it does limit how the testing group will implement its testing.

The two, high level ways of implementing testing are often in debate: Which is better, manual testing ("hand testing") or automated testing? Although there's no one right answer, the biggest difference between the two is time. Automated tests, in general, take much longer to create and maintain, but are very quick to run -- and you can run them over and over again. Manual tests, on the other hand, are cheap to create, take little to no time to maintain (it depends on implementation) but generally take a lot of tester time to run over and over again. Because our release is looming weeks away, and because this testing effort is primarily focused on this release (as opposed to future releases), the testing group will primarily focus on manual tests that save time upfront, rather than creating automated tests that may save time in the future.

There may be a chance, as well, that the testing group will create a small set of functional, automated tests to be used as "smoke tests" that can be run post-deployment. This may or may not happen, though, and depends primarily on time and stakeholder interest.

Process

As we all know, TOPP strives to be an agile (lowercase "a" agile) organization. We like to be responsive to change when it happens. The testing group, then, should be no different.

The Bach brothers -- James and Jon Bach -- have spent a lot of time coming up with a testing process that allows for this type of change. In fact, this process allows for a new testing mission every 90 minutes. They call it session-based test management.

I'm not going to go into too much detail here, but I do want to point you to two different papers written individually by the Bach brothers, as well as give you an overview of how this method fits within our organization.

• "Exploratory Testing Explained," by James Bach.
  
• "Session Based Test Management," by Jon Bach.
  

The Basic Idea

The basic idea behind session-based test management is simple: give testers time to test, and give them something to focus on. Using the SBTM lingo, the "time to test" is called a session, and the "something to focus on" is called a charter.

The session is important because it gives testers something to talk about. What did you find in this session? How did you use your time? What did you explore? It's a level of scope; it's manageable. They last roughly 90 minutes.

The charter is important because, as stated, it provides focus. It allows for coverage of the important items, but does not force the tester to perform any specific task. Like Christopher's Columbus' charter to find India, testers may find America instead, even if they were looking for something different. This is a good thing, and is embraced within this method of testing.

Debriefing

After each session, it is important to communicate what was found by the tester during the session. This generally ends up being a short meeting between the tester and the test manager, where the tester debriefs the test manager on what was found. This has three important consequences:

1. The test manager gains information about the state of the system that can be sent out to those involved.
2. The tester may learn from the test manager, or vis versa, about how to test a specific part of the system.
3. This gives the test manager information that can be used to create new charters, and help re-prioritize future tasks.

Bureaucracy & Record Keeping

The Bach brothers give a method of session-based test management that includes a lot of record keeping in order to manage the testing effort. Although I believe most of this record keeping is useful to the management process, our focus is specifically on the Woonerf release. This means a large amount of record keeping -- especially for a "first attack" at a structured testing group -- will only turn into unneeded overhead, getting in the way of our ultimate goal. If this testing group were to still be in effect after the Woonerf release, more record keeping may be needed; for now though, the only records officially kept will be those needed to successfully communicate the status of the system.

Strengths and Weaknesses, and TOPP

The particular strength of session-based test management is that it allows for agile testing in a way that's manageable. Test missions change every 90 minutes, allowing the testing group to switch focus quickly if an important task is brought to light. Its weakness, however, is that it is highly dependent on the test manager. The test manager is responsible for creating and assigning new charters, as well as communicating results to stakeholders.

For TOPP, this weakness is of considerable interest. I've been told that the test-manager may end up being the only person who will communicate bugs to developers. This means that, after debriefing -- and verifying that bugs are actually bugs -- the test manager will have to write those bugs into the bug tracking system him or herself. Although this method will maintain the anonymity of  testers (who, in this case, are also developers), it doesn't scale well. For a small team, this shouldn't be a huge problem, though it still may end up being too much for one person. Instead of making this the responsibility of the test manager, we should focus on verifying bugs in debriefing (this'll catch the "personal interest" and the "not really bug" bugs, removing need for anonymity), and then focus on teaching testers to write clear reproduction steps. Again, the goals of the testing group is short-term -- and so thinking about scale may be a bit out of scope -- but it may be interesting to throw around ideas in this area.

"Fluxicity" 

I'll be the first to say that I've never led a testing group in the real world, let alone tried session-based management. I'm using the Bach brothers' session-based test management as a model for manual testing, but how we do it may very well change as time goes on. I expect this whole process to be in flux until we find what works best for us, and I would appreciate any input you might have.

-- Tim