The Art of Software Testing
Chapter 7: Usability (User) Testing
Glen Myers
Wiley
Third Edition, 2012
An important category of Higher-Order test cases is one that attempts to find
human-factor, or usability, problems
Usability Testing Basics
List of questions you
might ask to derive testing considerations:
l. Has each user interface been tailored to the intelligence, educational
background, and environment of the end user?
2. Are the outputs of the program meaningful, appropriate for the typical user,
and devoid of computer gibberish?
3. Are the error messages straightforward or
does the user need a degree in Computer Science to comprehend them?
4. Does the total set of user interfaces exhibit
an underlying consistency and uniformity of syntax, conventions,
semantics, format, style, and abbreviations?
...
6. Does the system contain an excessive number of options, or options
that are unlikely to be used?
Intelligent menu system -- present those menu items that are most relevant
Later present those menu items that they frequently access
7. Does the system return some type of immediate acknowledgment for
all input?
Item can change color or a button object can depress
Processing time -- message should be displayed informing the
user of what is going on
8. Is the program easy to use?
Is the input case-sensitive
without making this fact clear to the user?
If a program requires
navigation through a series of menus or options, is it clear how to
return to the main menu?
Can the user easily move up or down one level?
Can an action be easily undone (when possible)?
9. Is the design conducive to user accuracy?
More important -- Does the design make it too easy for users to do
inappropriate things?
10. Are user actions easily repeated in later sessions?
Is the software design conducive to the user learning how to be
efficient in using the system?
Usability or user-based testing basically is a black-box testing technique
If users perceive, because of improper design, a cumbersome user
interface, or requirements missed or ignored, that a given application
does not perform according to its requirements, the development
process is viewed as a failure!
Usability Testing Process
Usability testing is more than simply seeking user opinions or high-level reactions to a software application
Any usability test should begin with a plan
Establish practical, real-world, repeatable exercises for each user to conduct
Design these testing scenarios to present the user with all aspects of the software
During each phase of the test, have observers document the user experience
as they perform each task
When the test is complete, conduct an interview with the user
Write down detailed instructions for user tests, to ensure
that each user starts with the same information, presented in the same
way
Test User Selection
A complete usability testing protocol typically involves multiple tests from
the same users, as well as tests from multiple users
Reason for multiple tests from same users --
One area we want to test is user recall, how
much of what a user learns about software operation is retained from session
to session
Software targeted for a specific end-user type or industry
should be tested by expert users, people already
familiar with this class of application in a real-world environment
Software with a more general target market --
Hallway testing, users
chosen for software testing are selected at random from people passing by
in the hallway
How Many Users Do You Need?
You need to find the right number of testers who can identify
the most errors for the least cost
Jakob Nielsen --
Approximately 83 percent of the errors are detected by only 5 testers
Approximately 98 percent of the errors are detected by 10 testers
Many question Nielsen's numbers and think more users are a better idea
But, it is clear that you do not need thousands ... or even hundreds ... of user testers
Among the considerations important to developers who are designing a
usability testing program are whether the number of users and their individual
orientations represent sufficiently the total population of potential
users
Data-Gathering Methods
Video recording a user test and using a think-aloud protocol can provide excellent
data on software usability and user perceptions about the application
Test participants describe out loud their task, what
they are thinking about the task, and whatever else comes to their mind
as they move through the testing scenario
Follow up with participants after
the test to get posttest comments, feelings, and observations
Remote user testing --
Application is installed
at the user's location where the software may ultimately be
applied
Remote testing has the advantage of placing the user in a familiar
environment, one in which the final application likely will be used, thus
removing the potential for external influences modifying test results
Usability Questionnaire
Develop questionnaires that generate responses that can be counted and
analyzed
yes/no answers
true/false answers
agree/disagree on a scale
Instead of asking "What is your opinion of the main menu
system," ask a series of questions that require an answer from
l to 5, where 5 is totally agree and l is totally disagree:
l. The main menu was easy to navigate
...
3. The screen design led me quickly to the correct software operational choices
...
5. The menu operations did not provide enough feedback to verify my choices
Good practice to ask the same question more
than once, but present it from the opposite perspective so that one elicits
a negative response and the other a positive one.