Summary of October KDE4 HIG Testing
This is a quick executive summary of the results of October’s user testing for the third iteration of the KDE4 Human Interface Guidelines (it is taking me too long to get caught up on KDE-related stuff to get a longer report out).
Methods
Four participants (all male) were recruited for two days of testing in two locations (Maryland and Virginia). Two were experienced software engineers, one of which was familiar with Qt. The other two had experience programming, however were not professionals. Most of the participants did not have interface design or implementation experience, however one participant did have moderate experience with interface guidelines.
The participants were asked to pretend they were building a dialog in a mail application, and find guidelines to interface elements they wished to use. This provides feedback of the location and labelling of certain topics or concepts. Afterwards, they were asked to rate their satisfaction and allowed to give any suggestions. This provides feedback of the participant’s overall experience.
Key Findings:
- Screenshots. Participants were consistent in stating the first thing they want is a screenshot to do a visual check. Then if they need more details, they’ll check the guidelines, and then theory/rationale if necessary.
- Less Concept, More Tech. Half of the participants had trouble with “Feedback” and looked for key words such as “alert box” or “messages”. Other misunderstandings suggest some of the labelling and organization is still too design-oriented and conceptual, and needs to be reexamined and reorganized. This issue has been the primary problem found in this architecture, as well as other popular guidelines documents, and the hardest to fix.
- Design HowTo. Several participants expressed interest in a “Introduction” section which teaches them about layout and design concepts in an instructional manner. This would replace a lot of the conceptual introduction material proposed, however addresses the same issues. Additionally, the Practical Examples are very important for this very reason; they provide solid, practical examples of good design — not conceptual gray matter.
- Examples. In many situations where the participant could not think of a keyword to look up a guideline for a higher-level concept, they went to the “Practical Examples” to see if there were any related examples they could draw assistance from. These examples are crucial for helping understanding of highly conceptual designs such as searching and file manipulation.
Study Limitations
The wiki format which was used for the test had possible usability issues. The design of the page may have competed with the content, and the length of the main page required users to scroll which may have led to missed topics. Overall, I do not think that any of the significant findings which I stated are effected by this issue.
Even though this was a test of the labelling and organization of content (information architecture), the lack of content may have effected user’s satisfaction or understanding of topics (since they did not have context attached to them). Additional content may have made it possible to gain more feedback from participants, but I do not believe it would have effected the current key findings. In fact, additional content may have hidden some important issues which were found.
Only four participants were tested, however they did have varying degrees of experience with programming, documentation, and user-centered design. It is possible that different issues could arise if more participants were tested, or the stated issues negated by other participant’s performance. However, the benefit of iterative design is that you test early and often. Testing a few users and finding consistent conceptual issues (such as labelling or overwhelming preference) is more beneficial than not testing during the development cycle and waiting until the end. These short usability “sprints” helps us identify any possible problems and correct them immediately, even if they are not statistically significant.
Next Steps
The next iteration should include more content and testing of that content. More information architecture issues can be followed up on then. I expect we should be able to add content and do additional testing spring 2007.
Special Thanks
I would like to thank the Reston Public Library and the University of Baltimore’s IAT department for use of their facilities.
All four participants opted to donate their participation stipend ($30 USD) back to the KDE e.V. Thanks again to
- John C.
- Nate K.
- Tim F.
- Varol O.
for your participation in this KDE activity and your gracious donations!
> The next iteration should include more content and testing of that content.
> More information architecture issues can be followed up on then. I expect
> we should be able to add content and do additional testing spring 2007.
Does that mean the HIG will be out when KDE4 is already developed a lot, or am I mistaken here?
Thanks a lot, I’m looking forward to the HIG, the process seams to be done really well.
Well unfortunately this is taking a really long time to write. There are only a few people working on this part time. We hope to have at least guidelines and screenshots available as soon as possible, however the actual document will follow much later.
The point of the document is to have a definitive source for all interface-related issues. There are a few major changes in KDE4 from 3 (such as label alignment), however most of it is just documenting common sense, and decisions which have been made to define the look, feel, and usability of the KDE environment.
The secondary goal of this document is to be a “living” example of good user-centered design, hence the focus on involving users (developers) and getting it right the first time :)
When you say that it takes a long time to write it, its more about knowing what to write then having people write it, right?
If not, then having a blog with an example chapter and several (example) requests for content will surely make programmers that know that component help out. (I hope ;)
TZ: yes, knowing what to write than actually writing it. We are trying to be very definitive about the guidelines, and that requires both looking in to what we have accepted as KDE standard and writing it down, and getting the right people together to propose a standard which hasn’t been defined. Imagine being given the task of describing something. How much detail must you go in to before it is clear to someone who has never seen it to be able to picture it?
Basically, its not all in my (or anyone else’s) head, ready to be typed out, we still have a lot of work to do defining KDE4 from the user’s perspective. The goal of this process (as opposed to write first, question later) is to reduce revisions which will cause a lot of work for developers, and to establish some kind of trust that these guidelines are it (so don’t wait for version 2 before implementing).