Reading Report 4 – “Usability Testing: What Have We Overlooked?”

Reading Report 4 – “Usability Testing: What Have We Overlooked?”

Citation:

Lindgaard, G., & Chattratichart, J. (2007). Usability Testing: What Have We Overlooked? CHI 2007, (pp. 1415-1424). Retrieved from http://www.itu.dk/~hulda/Usability/artikler/p1415-lindgaard.pdf

Article can be found here: http://www.itu.dk/~hulda/Usability/artikler/p1415-lindgaard.pdf

Key words:

Usability testing, Metrics, UEM (Usability Evaluation Method) , participant recruitment, number of users, number of tasks, recruitment of users

This article is about (main discussion):

The article explains research into usability testing to see if “The Magic Number 5” for users is accurate. The research is to find:

1. That there is a correlation between number of users and the proportion of problems found.
2. That there is a correlation between number of user tasks and the proportion of problems found.

In order to study these two ideas, the authors did not perform usability tests, rather they were allowed to use the raw data from a previous study from a previous study (CUE-4 from CHI 2003) into the number of users needed. They took data from nine of the usability teams. The nine teams all completed usability tests on the same hotel booking website, but the nine groups did not have the same user tasks or number of users. Lindgaard and Chattratichart analyzed the number of users, the number of user tasks and scenarios, and the number of problems that each group identified (p. 1417).

Upon completing their statistical analysis they found:

• The first hypothesis that there is a correlation between the number of users and the proportion of problems found was not supported.
• The second hypothesis that there is a correlation between the number of user tasks and the proportion of problems found was supported.

 The authors argue and state that:

• There has been too great of a focus in ID on the number of users when the focus should really be on the number of user tasks and also the quality of participants.
• The Magic Number of 5 does not hold. In this study, “the percentage of problems found by the nine teams ranged from 7% to 43% - nowhere near the predicted 85%. The argument [the 5 users will find 85% of the problems] is therefore not upheld by the present data” (p. 1422)
• Giving users a persona to imagine during the tasks is helpful because “[t]he persona might have helped their test users place themselves in the real users’ shoes and hence carry out the required tasks the way real users would do. As a result, Team S [the only team with a persona] performed better than expected” (p. 1423)
• Rather than the number of users, the focus should instead be on “careful participant recruitment […]. It [also] pays off to give many sets of user tasks to a small number of users in a usability test rather than giving many users the same limited set of user tasks in a usability test” (p. 1423).

The authors conclude that:

Less resources and research should be spent on studying the number of users to use for usability testing. Instead the focus of further research should be on “the role of user tasks on improving usability testing approaches as well as into the importance of recruitment of test users” (p. 1424).

References: 16, (1981-2004)

Relating this to our project:

• When creating our usability tests, we should focus on increasing the number of user tasks
• The quality of the user tasks and scenarios must also be a consideration
• It will be more fruitful to give a higher quality usability test to a small number of users
• Care and attention must be taken to ensure that we have a good variety of users. The users should have varying levels of computer savvy and from different age groups. We should not just focus on using one group of users because they may all find different errors or problems. For example, a more tech savvy user may uncover issues with the usability than a less skilled user. They may approach the slide collection in a much different way
• It may be beneficial to have the users take on a persona while completing the user tasks