The Art of Software Testing

Chapter 7: Usability (User) Testing


Glen Myers
Wiley
Third Edition, 2012





  • An important category of Higher-Order test cases is one that attempts to find human-factor, or usability, problems

    Usability Testing Basics

  • List of questions you might ask to derive testing considerations:

    l. Has each user interface been tailored to the intelligence, educational background, and environment of the end user?

    2. Are the outputs of the program meaningful, appropriate for the typical user, and devoid of computer gibberish?

    3. Are the error messages straightforward or does the user need a degree in Computer Science to comprehend them?

    4. Does the total set of user interfaces exhibit an underlying consistency and uniformity of syntax, conventions, semantics, format, style, and abbreviations?

    ...

    6. Does the system contain an excessive number of options, or options that are unlikely to be used?

    Intelligent menu system -- present those menu items that are most relevant
    Later present those menu items that they frequently access

    7. Does the system return some type of immediate acknowledgment for all input?

    Item can change color or a button object can depress

    Processing time -- message should be displayed informing the user of what is going on

    8. Is the program easy to use?
    Is the input case-sensitive without making this fact clear to the user?
    If a program requires navigation through a series of menus or options, is it clear how to return to the main menu?
    Can the user easily move up or down one level?
    Can an action be easily undone (when possible)?

    9. Is the design conducive to user accuracy?
    More important -- Does the design make it too easy for users to do inappropriate things?

    10. Are user actions easily repeated in later sessions?
    Is the software design conducive to the user learning how to be efficient in using the system?

  • Usability or user-based testing basically is a black-box testing technique

  • If users perceive, because of improper design, a cumbersome user interface, or requirements missed or ignored, that a given application does not perform according to its requirements, the development process is viewed as a failure!

    Usability Testing Process

  • Usability testing is more than simply seeking user opinions or high-level reactions to a software application

  • Any usability test should begin with a plan

  • Establish practical, real-world, repeatable exercises for each user to conduct

  • Design these testing scenarios to present the user with all aspects of the software

  • During each phase of the test, have observers document the user experience as they perform each task

  • When the test is complete, conduct an interview with the user

  • Write down detailed instructions for user tests, to ensure that each user starts with the same information, presented in the same way

    Test User Selection

  • A complete usability testing protocol typically involves multiple tests from the same users, as well as tests from multiple users

  • Reason for multiple tests from same users --
    One area we want to test is user recall, how much of what a user learns about software operation is retained from session to session

  • Software targeted for a specific end-user type or industry should be tested by expert users, people already familiar with this class of application in a real-world environment

  • Software with a more general target market --
    Hallway testing, users chosen for software testing are selected at random from people passing by in the hallway

    How Many Users Do You Need?

  • You need to find the right number of testers who can identify the most errors for the least cost

  • Jakob Nielsen --

  • Approximately 83 percent of the errors are detected by only 5 testers

  • Approximately 98 percent of the errors are detected by 10 testers

  • Many question Nielsen's numbers and think more users are a better idea

  • But, it is clear that you do not need thousands ... or even hundreds ... of user testers

  • Among the considerations important to developers who are designing a usability testing program are whether the number of users and their individual orientations represent sufficiently the total population of potential users

    Data-Gathering Methods

  • Video recording a user test and using a think-aloud protocol can provide excellent data on software usability and user perceptions about the application

  • Test participants describe out loud their task, what they are thinking about the task, and whatever else comes to their mind as they move through the testing scenario

  • Follow up with participants after the test to get posttest comments, feelings, and observations

  • Remote user testing --
    Application is installed at the user's location where the software may ultimately be applied
    Remote testing has the advantage of placing the user in a familiar environment, one in which the final application likely will be used, thus removing the potential for external influences modifying test results

    Usability Questionnaire

  • Develop questionnaires that generate responses that can be counted and analyzed

    yes/no answers
    true/false answers
    agree/disagree on a scale

  • Instead of asking "What is your opinion of the main menu system," ask a series of questions that require an answer from l to 5, where 5 is totally agree and l is totally disagree:

    l. The main menu was easy to navigate
    ...
    3. The screen design led me quickly to the correct software operational choices
    ...
    5. The menu operations did not provide enough feedback to verify my choices

  • Good practice to ask the same question more than once, but present it from the opposite perspective so that one elicits a negative response and the other a positive one.