Why aren’t there more user-based usability testing in OSS? IMHO it is because of the time and money costs.
Even after the planets have aligned and we have a usability specialist with the skills and an open source project committed to doing something with the results, there are still difficulties in Getting Things Done. The most annoying and unavoidable one is the cost (monetary) for conducting the test. Forgoing the moderator and second chair observer as loyal open source advocates and willing volunteers, there are still the matters of participant stipends (monetary or gift) and laboratory setting. Even when participants are willing to give up an hour of their time, the testing environment is the hardest part to acquire on a budget.
To properly conduct a user-based usability test, you need one of three environments:
- A computer which can record audio, screen, and external video for participant’s face (referred to as Superman testing because the moderator is on their own and must review the tapes later for analysis)
- A computer which a second chair observer can view the participant — either through one-sided glass or remotely in real time (usually includes video/audio recording as well, but without such can be forgiven if the notes and analysis from the second chair observer are excellent)
- Remote usability testing software which shares audio (via microphone or conference calling), screen sharing and recording, and allow observers (not as high quality of feedback, but is practical in early-cycle testing and shoe-string budgets)
Both of the laboratory environments usually cost money to rent. Remote usability testing has some cost associated with it, but is often much less expensive.
Remote testing can be conducted without a laboratory setting using a variety of software, most popularly with Webinar software that many companies own to conduct online meetings. Remote testing is commonly conducted by using GoToMeeting, Connect (née Breeze), or UserVue. All of these solutions cost money (some are cheaper than others) and none are Free/Open Source, let alone run on Linux.
In the past, I have used VMWare in Windows so I could run Linux and record screen, audio, video, and mouse clicks with Morae. I have not yet found a solution I could use for remote testing, regardless of license or cost. I would really like to standardize an easy to set up remote usability testing environment that can be used in both face-to-face studies and remote tests. It is acceptable if a lot of configuration up front must take place, but participant technical requirements (hardware, installation, configuration) must be kept at a minimum.
(Remote) usability testing environment requirements:
- Screen and mouse sharing for both moderator and participant 1
- Audio streaming of both moderator and participant 1
- Ability for second chair observer to watch/hear moderator and participant 1
- Screen capture/recording 2
- Audio recording of both moderator and participant 2
- Fullscreen mode for testing environment 3
- Participant face (web camera) recording 3
- Mouse highlighting in post-production 3
- Click highlighting in post-production 3
- Picture in picture (participant in corner of screen) in post-production 3
1: Absolutely required, will still require a second chair observer
2: Minimum requirements for a serious remote user-based testing study (including 1)
3: Would be really nice to have and make reviewing and creating highlights videos much easier (including 1 & 2)
Are there any suggestions on how to accomplish this? The floor is open, let me hear your ideas!
seele :: Dec.06.2007 ::
General, Open Source, Usability ::
8 Comments »