Discuss: Open Source Usability Testing Solutions
Why aren’t there more user-based usability testing in OSS? IMHO it is because of the time and money costs.
Even after the planets have aligned and we have a usability specialist with the skills and an open source project committed to doing something with the results, there are still difficulties in Getting Things Done. The most annoying and unavoidable one is the cost (monetary) for conducting the test. Forgoing the moderator and second chair observer as loyal open source advocates and willing volunteers, there are still the matters of participant stipends (monetary or gift) and laboratory setting. Even when participants are willing to give up an hour of their time, the testing environment is the hardest part to acquire on a budget.
To properly conduct a user-based usability test, you need one of three environments:
- A computer which can record audio, screen, and external video for participant’s face (referred to as Superman testing because the moderator is on their own and must review the tapes later for analysis)
- A computer which a second chair observer can view the participant — either through one-sided glass or remotely in real time (usually includes video/audio recording as well, but without such can be forgiven if the notes and analysis from the second chair observer are excellent)
- Remote usability testing software which shares audio (via microphone or conference calling), screen sharing and recording, and allow observers (not as high quality of feedback, but is practical in early-cycle testing and shoe-string budgets)
Both of the laboratory environments usually cost money to rent. Remote usability testing has some cost associated with it, but is often much less expensive.
Remote testing can be conducted without a laboratory setting using a variety of software, most popularly with Webinar software that many companies own to conduct online meetings. Remote testing is commonly conducted by using GoToMeeting, Connect (née Breeze), or UserVue. All of these solutions cost money (some are cheaper than others) and none are Free/Open Source, let alone run on Linux.
In the past, I have used VMWare in Windows so I could run Linux and record screen, audio, video, and mouse clicks with Morae. I have not yet found a solution I could use for remote testing, regardless of license or cost. I would really like to standardize an easy to set up remote usability testing environment that can be used in both face-to-face studies and remote tests. It is acceptable if a lot of configuration up front must take place, but participant technical requirements (hardware, installation, configuration) must be kept at a minimum.
(Remote) usability testing environment requirements:
- Screen and mouse sharing for both moderator and participant 1
- Audio streaming of both moderator and participant 1
- Ability for second chair observer to watch/hear moderator and participant 1
- Screen capture/recording 2
- Audio recording of both moderator and participant 2
- Fullscreen mode for testing environment 3
- Participant face (web camera) recording 3
- Mouse highlighting in post-production 3
- Click highlighting in post-production 3
- Picture in picture (participant in corner of screen) in post-production 3
1: Absolutely required, will still require a second chair observer
2: Minimum requirements for a serious remote user-based testing study (including 1)
3: Would be really nice to have and make reviewing and creating highlights videos much easier (including 1 & 2)
Are there any suggestions on how to accomplish this? The floor is open, let me hear your ideas!
seele :: Dec.06.2007 :: General, Open Source, Usability :: 8 Comments »
If I understand correctly, this is an honest and timely call for anyone who is/wants to contribute to the world of FOSS significantly to create a multi-platform program which handles all of the above. Personally, though I am not a programmer, this doesn’t look too difficult. If I understand this correcty, are there any takers?
You missed one thing from your list of requirements: a thick skin for dealing with the Dot commenters who will tell you that despite all this time and effort your tested solution just happens to be the worst possible replacement for what they use already. ;)
Me, bitter? Never.
Regarding “Screen capture/recording”: I’ve been looking at hardware devices capable of capturing VGA output from a computer for recording presentation with video/animations at next akademy. The price might be a bit to steep to acquire it for a annual event. If we use it for usability testing that cost might be justified.
Bart
hey
http://www.yugma.com/ would be my suggestion.
It’s free, multi-platform, has some sort of skype version, and version that requires no software installed.
You need to subscribe to premium to get some of the features you want permanently tho.
I very much wish you success with this project! With a guidebook and free software, I would think educators would want to volunteer usability testing to make their own labs more productive.
I don’t have any real solutions, but seems that VNC would provide a good chunk of what your looking for. Shared sessions, full screen, low overhead.
Recording VNC:
http://www.cs.ubc.ca/~bsd/vncrecording.html
http://emergent.unpythonic.net/01196105360
http://www.unixuser.org/~euske/vnc2swf/
Webcam software:
http://www.lavrsen.dk/twiki/bin/view/Motion/WebHome
http://www.lavrsen.dk/twiki/bin/view/Motion/SoundAudioRecording
How I would see this working is a webcam pointed at both the participant and their screen, so you could sync up their facial expressions with the output from VNC.
I’d think that you’d want the webcam software running on separate machine(s) so the high bandwidth doesn’t interfere with the user’s experience. Same idea may go towards the VNC recording software.
So, like so many things with FOSS, my idea involves several pieces of software, not one nice rolled up package. Good luck with this!
Has anyone spoken to any of the companies that donate time and money to open source (such as IBM)? Has there been any discussion to at least find some methods to have usability testing for the bigger projects if not to develop usability testing software?
@Bart
I have recorded screen video using AVI out to a DV recorder, but only in a lab setting. I’m not sure how it could be used for remote testing efficiently because raw video is expensive (bytes).
@Samu
I haven’t used Yugma before, but it looks similar to GoToMeeting. Although Yumga allows Linux users to observe, they cannot host, which is essential for testing a Linux application. However, they do mention that Linux hosting is in the future and at $10USD a month, it could be a decent option (although, not a FLOSS option).
@jimcooncat
Thanks for the VNC recording links. Like you said, there are lots of pieces that can fulfill each of my requirements, but no one application can do it.
@Steve
There are a few organizations who have done their own testing and reported back to the community. Relevantive through OpenUsability have conducted a number of usability tests, I have done a few for Kubuntu and KDe, and I believe Suse has done testing as well. It would be great to have a sponsor to support lab and participant costs.
This may be a bit offtopic since you’re asking for remote usability testing tools, but anyway: When doing usability testing on a shoestring budget, I’ve found that a combination of laptop + cheap table microphone + Windows (ugh!) + CamStudio (GPL) makes a great toolkit.
The software makes a screen recording with audio from the mike. You make some notes during the session about the reactions of the user (since there’s no face camera to record that) and if necessary you can analyze the screencast recording afterwards.
No need for an expensive lab, you can test anywhere (I’ve done tests at work, in people’s homes, in a cafeteria back room etc.). The downside is that the test user can see the reactions of the observer unlike in a real lab, but if you’re willing to sacrifice that, the setup works very well.
You still need VMware or something similar for testing Linux apps. I’ve mostly done web applications where the OS doesn’t matter. I haven’t found a Linux app that would make screencasts as effortlessly as CamStudio (synchronized recording of screen and audio, keyboard shortcuts and moderate use of resources would be the main requirements) but some apps seem to be getting there so maybe I’ll switch in the future.
http://camstudio.org/