Project Home   |  Planning Document   |  Guidelines   | Task Test  |  Interview Document |  Closing survey

1. Selecting users

• A call for volunteers can be sent out by email and/or posted on various online job and classified listings (beware of the number of replies you might get).
• Participants are screen and selected in order to provide a varied group of people. The use of an online screener test can reduce greatly the time it will take to select participants. Follow-up on the phone also helps.
  
• Efforts should be made to cover at least half of the project's original user profiles. Participants should not have been involved in the design or creation of the website and, in most cases, should have as little prior knowledge of the site as possible.
• Decide what the user's minimal tech knowledge should be and look for this in the screener.
  
• 6-8 testers is usually enough to cover most problems. Even 5 will often do.
  
• In some case, financial compensation can be offered (50-100$). Some people will argue that incentives can be problematic when trying to get real input from users.
  
• Interviews can happen in the user's regular work space or at the organization's offices (the rest of this info will assume the testing happens in your office.
• Unlike with regular observation testing, in Co-discovery testing, participants come in as a pair. This can help users better express what they are trying to do and can better reflect certain realities of using an online products (if you don't know, you ask your desk neighbor first)

2. Interview details

Welcome, thanks, drinks?
If there is an opening questionnaire, make them fill it
Describe the lab, the cameras, the people
Describe how the session will unfold, the schedule
We are testing the site, not them
Fill out release and sign it
Describe "think aloud" procedure
Practice with a stapler
Ask if there are any questions
Hand participant the written questions
Read the test and questions aloud for them as you go. They should only refer to  the written document themselves if it helps them.
Thank you for doing this and remember, think aloud and no right or wrong answers
Start the interview

At the end
Thanks
Fill out the closing questionnaire
Sign receipt of money and give out compensation
Bye Bye

3. Things to measure:

• Effectiveness
• Was a task completed or not
• Was the results of task good or not
  
• Performance/Efficency
• time it takes to accomplish tasks
• number of errors and mistakes made
• unproductive tasks and dead-ends
• count clicks (when possible)
• Satisfaction
• subjective reactions
  
• gather comments and ratings
  
• ask about the expected vs. the actual experience
• observe reactions of confusion, frustration, surprise and satisfaction

4. User research Team

Person 1: Leader, greeter, briefer. Narrates the test.
Person 2: Note taker #1 and test operator, if needed (load websites, setup user, load questionnaires etc...)
Person 3: Note taker #2 and video recording
+ have someone on tech support available around in case something goes wrong, including filming.


Setup
Closed room, isolated
1 computer with keyboard and mouse
Choose a large screen or use a small projector to faciliate note taking and observation
If filming: camera, tripod, avoid lighting or harsh conditions for users

5. Schedule, general

Book participants every hour. Hourly breakdown goes as follows:
00-10: participant arrives, welcome, something to drink, sign releases
10-45: actual test
45-50: closing questionnaire (written or on screen), pay (if there is), good bye
50-60: team cleans-up notes, preps for next participant

OR option B, One interview every 1:15hours
00-15: participant arrives, welcome, something to drink, sign releases
15-50: actual test
50-60: closing questionnaire (written or on screen), pay (if there is), good bye
00-15: team cleans-up notes, preps for next participant

6. Note on writing tests

Use previously written user stories as a starting point for the task.
An average of 4 or 5 general task sets is enough and should take about 30-40 minutes to complete.
Test the tests first with people in the office.
Do not teach participants about the site through the instructions.

From some site: If you're concerned that graphic design might be affecting company image, don't ask "What do you think of that red logo?" Instead ask something like, "How do you feel about this company?" or "What kind of company do you think this is? What words would you use to describe the people behind this site?"

7. Recommendations and results

Provide more or less formal report document with details for each element that was meant to be tested.

Classify usability problems (from norman/nielsen i think)

• Catastrophy: user could not complete the task properly
• Serious: users where slowed down significantly but managed to complete the task afterall
• Cosmetic: users where slightly slowed down or annoyed or expressed dislike