obso1337.org

A while back, there was some discussion about how Ubuntu LoCos could get more involved in usability, such as through testing. So for the past few months, I’ve been talking with some Ubuntu LoCo members on how we can Get Things Done.

Usability testing is when you ask a study participant (who has similar characteristics as your target users) to complete a series of tasks using your product. The goal is to get feedback about the product so you can make it better. There are various types of testing protocols, but the one I use the most often as part of the design process is called an interrupted think aloud protocol (TAP). The participant is given a scenario and asked to complete a set of tasks. They may work quietly, but talk aloud if they like or ask questions. The moderator may or may not answer their questions in order to see how the participant attempts to solve the problem. The moderator may also interrupt the participant during the task scenario in order to discuss a particular choice or action.

This isn’t performance-based testing because the participant is distracted by moderator’s presence, periodic interruptions, and thinking/talking aloud, and is more useful than a pure TAP because the participant only speaks when they need to or asked to. When participants are required to speak through every thought and action, it can be very distracting to the task and affect the participant’s problem solving skills and performance on the task.

But what about actually running usability tests? You can teach a monkey to run a test, but that doesn’t mean they will be any good at it. The main problems which came to mind were moderator skill and analysis of usability issues. Two of the most difficult skills to master as a test moderator are establishing a rapport with the participant, and knowing when to help a participant or let them struggle through a task.

The first moderating skill problem, rapport, can be solved in a simple recruiting strategy. In what is called a “friends and family” recruit, LoCo members could find people they know who have certain characteristics (such as a certain age, computer experience, or education) and invite them to participate. The existing relationship between the moderator and participant allows the participant to speak and act freely instead of feeling like they are under a microscope. However, this existing relationship may complicate the second moderating skill problem, knowing when to help. Because the moderator has an established relationship with the participant, they may be more inclined to help the participant at the first signs of trouble rather than allowing the participant to figure it out for themselves. Hopefully careful training will help mitigate this effect.

The identification and analysis of usability issues is a tough problem to solve. LoCo members may be able to report what the user said or did, but not necessarily why. As a moderator, they will be busy with the participant and may not be able to take good notes or they might miss something important if they do. Additionally, without rich context, even reporting what the user said or did may not be enough information for a designer to diagnose a problem. This is where the “second chair” comes in.

A second chair is a second observer, usually “behind the glass”, who can concentrate on the participant and the design. Usually when I run usability testing in my company, I am the second chair. As the person who will be identifying and analysing issues, it is more important for me to watch what the user is doing and take good notes than for me to walk the participant through the activity. So in order to gather useful feedback, the LoCo running the usability test would need to have someone experienced in design to take notes for them. Unfortunately, the requirement of having an experienced designer as second chair limits the activity’s autonomy and creates requirements which not all LoCos would be able to fulfill. But hey, it’s better than nothing, right?

Last Saturday, I met up with members of the Washington, D.C. Loco and ran a pilot usability study. The purpose of the pilot was to test the lab location, the computers used for testing, and the testing protocol. In addition to practicing moderating and running through the protocol on ourselves, we had two library patrons and a LoCo member’s “friend and family” participate. Since this was a pilot test, I will not be discussing the results only the logistics and lessons learned.

The LoCo has a small computer lab (3 computers) running Ubuntu Hardy 8.04 in a conference/storage room in a city library. The goal was to provide an computing resource for library visitors who are interested in learning more about open source software. Over time, the lab sessions turned in to general computer and Internet training for library patrons, with a few actually bringing in their Windows computers and asking for help. The LoCo members were happy to help anyone who came in, but their message of open source software was no longer getting through.

The usability testing activities were seen to be a way to bring the “open source” back in to the lab. It would encourage more LoCo members to stop by the lab by participating in the activity, and it would also expose library patrons to open source software through volunteering as a participant while someone fixes their computer.

I wanted the testing sessions to be short (15-20 minutes) for several reasons. One, being a participant in usability testing can be tiring, even if it is a simple activity. Also, if multiple people show up, they become a distraction if too many people have to wait. If there are no other participants waiting, then an additional small activity could be added to the current participant’s session without making it unreasonably long. Finally, we will be more flexible and get more out testing if we focus on pieces of problems instead of trying to gather too much data.

For this pilot test, we focused on the Digital Camera experience. Participants were asked to plugin a camera, download photos, manipulate one, and share it. A digital camera with photos was provided. This activity tested the hardware notification dialog, and F-Spot functionality. (We had one participant use the GIMP to do some simple editing, but it was too heavy of a tool for simple editing activities.) The testing environment was a PIII with 512MB RAM and a 17″ monitor running at 1280×1024. In addition to testing the protocol with LoCo members, we had 3 “real” participants.

• Participant 1 was about 45-55, owned a computer and had some computer skills, owned a digital camera but only downloaded photos from the camera and never really edited them.
• Participant 2 was about 55-65, owned a computer and had some computer skills, didn’t own a digital camera but was thinking about buying one.
• Participant 3 was about 35-45, owned a computer and had good computer skills, owned a digital camera and liked to use “scrapbook” software for editing photos.

This was a pretty good sampling from audiences we don’t typically get feedback from (the highly adept and opinionated). We expect that participants recruited by DC LoCo members will have average to above average computer experience and walk-in library patrons will have average to below-average computer experience.

The pilot was a success and I am confident we will be able to run several future activities and gather lots of usability feedback. Tentatively, we are planning to run the usability activity once a month for a few months to see how it goes and if we can continue to get participant recruiting support from our LoCo members. We will advertise to our LoCo members and encourage them to pick a date to bring a friend or family member, as well as encourage library patrons who stop by to participate. I also hope that we can do similar activities in LoCos around the world (if we can find usability engineers/designers to take notes) and at conferences.

Some additional notes I’ve made to improve for the next session:

• Activities need to be 15-20 minutes, especially for walk-in library patrons. Having too many people in the lab is distracting and some participants may not have an hour to commit.
• We shouldn’t expect more than 1-2 walk-in library patrons and should try to encourage an additional 3-4 LoCo members to bring a friend or family by. This will give us 4-6 participants per session which I think is a good target for a recurring activity.
• A list of scenarios/activities would be useful so we can use the most appropriate scenario for the type of participant we have or allow participants with time to complete more than one activity.
• A standardized participant survey would help use gather data and better classify the participant in to a user group and match the data from one study to another.

A big thanks to Washington, DC Ubuntu LoCoers Kevin Cole and Sanjay Jain for setting up and maintaining the Cleveland Park Library Ubuntu Lab. We would also like to thank Canonical for provided some nice Ubuntu pens, stickers, mousepads, and keychains as thank you gifts for our participants.