The Art of Software Testing

Chapter 11: Mobile Application Testing


Glen Myers
Wiley
Third Edition, 2012



Some of this lesson was written by and is used with permission of Purdue University Computer Science students Raghav Shankar and Kamran Pirwani.

Introduction

  • Handheld mobile and smartphone devices have significantly affected the way software developers and testers do their jobs

  • Most software testing professionals find testing mobile applications very challenging -- more so than almost any other software types or platforms

  • The devices and mobile environment add many variables and complexities that may skew or mask problems in your application, which makes designing a robust test plan difficult

  • Need to consider network performance and reliability, consistent user interfaces, device diversity, and limited resource platforms

  • Customers demand a good user experience. If they are not satisfied with the product, they can provide bad reviews. In most cases, these reviews are public and easily accessible. Hence, the quality of a mobile application is critical for its success.

    Mobile Environment

  • Mobile device is one that has the capability to run network-based applications over a cellular or satellite data link

  • Design your mobile application with the expectation that it will use relatively-slow, comparatively-unreliable data links

  • Focus is on the challenges associated with applications running on cellular data networks

  • Identify software problems when the carrier's network is slow or unavailable

  • To create successful test plans, you must consider the numerous devices in the marketplace, the varying capabilities of each, and how the user interacts with the devices

    Testing Challenges

  • Shear combination of device types, operating systems, user input methods, and network concerns
    Trade-offs must be balanced with time, financial, and labor resources
    Arrive at an economical test plan that detects most bugs in a reasonable time frame

    Mobile Device Diversity

  • The ever-expanding diversity of devices presents an often-underestimated and significant testing challenge to someone new to mobile application testing

  • Suppose a manufacturer develops a new method of text input via the touch screen for its phones
    Design a test to determine whether the device's new input method breaks your application
    If it does, can you fix your application without breaking support for other devices?

  • Mobile application testing is difficult due to compatibility issues since a mobile application can be deployed across devices which have different...

  • Operating systems like iOS, Android, Windows, etc.

  • Versions of an operating system such as iOS 7.x, iOS 8.x, etc.

  • Manufacturers like Samsung, Motorola, HTC

  • Based on the manufacturer, the device can have different
    CPU specifications
    Operating Systems
    Memory specifications
    Screen resolutions

  • There exist a variety of screen sizes for mobile devices
    Adapting applications to various screen sizes, layout, and configuration is a challenging task

  • Many mobile products involve a significant hardware element such as scanners, mobile telephony, GPS and position based devices, telemetry, etc.

  • This device diversity places additional demands on the tester, particularly in terms of isolating a bug to software or to hardware

  • As a result, a software quality team cannot guarantee that if an application works on one device, it will work perfectly (or even acceptably) on all other devices

  • Do as much testing as you can with emulators

  • Using emulators is great way to start, but ultimately you will need to test mobile devices on actual carrier networks

    Scripting vs Manual Testing

  • Automated testing of desktop and web applications is accomplished by creating script files

  • But, mobile devices do not allow you to load automated, repeatable scripts onto the device
    Test personnel manually execute all scripts

  • Manual testing is error prone
    Unfortunately, it is unavoidable when testing mobile applications on mobile devices

  • Mobile devices, unlike web applications, have a variety of features which depend on the surrounding environment including cellular network, GPS navigation, etc.

  • Automating the testing process by placing the mobile device in a lab may not be sufficient

  • Test Automation also becomes difficult due to device diversity

  • Different devices may have their own keycodes, keystrokes, input methods, and display structure

    Testing Approaches

  • Testing your application on the carrier's network is extremely important
    You want to find problems related to spotty coverage or sudden loss of connectivity

  • What happens when coverage is suddenly restored after an interruption during a transaction?
    Does a purchase occur twice?
    Check for bugs related to handling session reinitialization and data corruption

  • When dealing with cellular networks, you should expect to lose connectivity

  • Test applications in different locations which have low/high population density and good/bad cell signals

  • Measure amount of network used by the application

  • A test case specific to mobile testing is how your application handles incoming voice calls and text messages
    Chances are end users will want to suspend your application, or run it in the background, while they answer the phone or read and respond to the text message

    Testing with Mobile Devices

  • Only by testing with the device can you experience its nuances and get a true feel for the user's experience

  • Testing the reliability of a carrier's network and determining the effect of an incoming call or text message are obvious examples

  • On a mobile device you also can evaluate how your application behaves
    Does it load fast and run at an acceptable speed?
    Does it look okay?
    Is the UI consistent across your target devices?
    Last but not least, you can determine device-specific bugs
    This is almost impossible with an emulator
    If you do find a device-specific bug, the challenge is to fix it without breaking compatibility with other devices

  • Testing with mobile devices is (at least right now) a manual process

  • If your application is running slowly, you do not know whether it is the carrier's network, your application, or a combination
    Only by trial and error can you identify problems

  • Must ensure that mobile apps are working on all type of devices (Smartphones, Tablets, etc.)

  • Compatability with major brands (various models from Samsung, Sony, Nokia, HTC, Apple, etc.) and on all the platforms (iOS, Android, Windows, etc.)

  • New devices are hitting the market so quickly that it may be impossible to cover all the major devices

    Complex User Interaction Scenarios

  • Desktop applications limit user interaction to using the mouse

  • More complex scenarios can occur while using mobile applications

  • Interactions through complex gestures, eye movement, direction, tilt, movement, acceleration, GPS coordinates, surroundings, sound, etc.

  • Need to ensure that the app works as expected when the user interacts with the app in different ways

    Testing Resource Utilization of Mobile Applications

  • When developing a desktop application, an inefficient algorithm is not necessarily a serious issue

  • Limited resources on mobile devices make it important to have efficient algorithms

  • Good testing approaches include...
    Testing for average cellular network/CPU/Memory/Thread usage
    Excess usage of resources will limit the battery life of the mobile device
    Ideal testing strategies include testing common day-to-day functionalities and measuring resource utilization

    Testing with Emulators

  • Usually the most practical and cost-effective, and it even has some advantages

  • Emulators allow for inexpensive and quick functional testing of your application

  • Emulators run on computers with more resources, such as faster CPUs and more memory
    Fast response times during testing enables you to complete tests more quickly

  • Most emulators employ high-level scripting languages
    You can create consistent, automated tests, which are less error prone and quicker than manual testing

  • Automated scripting also allows for easier and faster regression testing

  • Do not rule out using emulators for as much of your testing as possible
    Cost-effective and efficient way to eliminate most of your bugs

  • Emulators are available for variety of systems -- Android, Windows Phone, iPhone etc.