The Art of Software Testing
Chapter 11: Mobile Application Testing
Glen Myers
Wiley
Third Edition, 2012
Some of this lesson was written by and is used with permission of
Purdue University Computer Science students Raghav Shankar and Kamran
Pirwani.
Introduction
Handheld mobile and smartphone devices have significantly affected the
way software developers and testers do their jobs
Most software testing professionals find testing mobile applications very
challenging -- more so than almost any other software types or platforms
The devices and mobile environment add many
variables and complexities that may skew or mask problems in your
application, which makes designing a robust test plan difficult
Need to consider network performance and reliability, consistent user
interfaces, device diversity, and limited resource
platforms
Customers demand a good user experience. If they are not satisfied
with the product, they can provide bad reviews. In most cases, these
reviews are public and easily accessible. Hence, the quality of a
mobile application is critical for its success.
Mobile Environment
Mobile device is one that has the capability to run network-based
applications over a cellular or satellite data link
Design your mobile application with the expectation that it will use relatively-slow,
comparatively-unreliable data links
Focus is on the challenges associated with applications running on cellular data networks
Identify software problems when the carrier's network is slow or unavailable
To create successful test plans, you must consider the numerous
devices in the marketplace, the varying capabilities of each, and how
the user interacts with the devices
Testing Challenges
Shear combination
of device types, operating systems, user input methods, and network concerns
Trade-offs must be balanced with time, financial, and labor resources
Arrive at an economical test plan that detects most bugs in a reasonable time frame
Mobile Device Diversity
The ever-expanding diversity of devices presents an often-underestimated
and significant testing challenge to someone new to mobile application
testing
Suppose a manufacturer develops a new method of text input via the touch
screen for its phones
Design a test to determine whether the device's new input method breaks your application
If it does, can you fix your application without breaking
support for other devices?
Mobile application testing is difficult due to compatibility issues
since a mobile application can be deployed across devices which have
different...
Operating systems like iOS, Android, Windows, etc.
Versions of an operating system such as iOS 7.x, iOS 8.x, etc.
Manufacturers like Samsung, Motorola, HTC
Based on the manufacturer, the device can have different
CPU specifications
Operating Systems
Memory specifications
Screen resolutions
There exist a variety of screen sizes for mobile devices
Adapting applications to various screen sizes, layout, and
configuration is a challenging task
Many mobile products involve a significant hardware element such as
scanners, mobile telephony, GPS and position based devices, telemetry,
etc.
This device diversity places additional demands on the tester,
particularly in terms of isolating a bug to software or to hardware
As a result, a software quality team cannot guarantee that if an
application works on one device, it will work perfectly (or even
acceptably) on all other devices
Do as much testing as you can with emulators
Using emulators is great way to start, but ultimately you will need to
test mobile devices on actual carrier networks
Scripting vs Manual Testing
Automated testing of desktop and web applications is accomplished by
creating script files
But, mobile devices do not allow you to load automated, repeatable scripts
onto the device
Test personnel manually execute all scripts
Manual testing is error prone
Unfortunately, it is unavoidable when testing mobile applications on mobile
devices
Mobile devices, unlike web applications, have a variety of features
which depend on the surrounding environment including cellular
network, GPS navigation, etc.
Automating the testing process by placing the mobile device in a lab
may not be sufficient
Test Automation also becomes difficult due to device diversity
Different devices may have their own keycodes, keystrokes, input
methods, and display structure
Testing Approaches
Testing your application on the carrier's network is extremely important
You want to find problems related to spotty coverage or sudden loss of connectivity
What happens when coverage is suddenly restored after an interruption
during a transaction?
Does a purchase occur twice?
Check for bugs related to handling session reinitialization and data corruption
When dealing with cellular networks, you should expect to lose connectivity
Test applications in different locations which have low/high
population density and good/bad cell signals
Measure amount of network used by the application
A test case specific to mobile testing is how your application handles incoming voice calls and text messages
Chances are end users will want to suspend your application, or run it
in the background, while they answer the phone or read and respond to
the text message
Testing with Mobile Devices
Only by testing with the device can you experience its nuances and get a true feel for the user's experience
Testing the reliability of a carrier's network and determining the effect of an incoming call or text message are obvious examples
On a mobile device you also can evaluate how your application behaves
Does it load fast and run at an acceptable speed?
Does it look okay?
Is the UI consistent across your target devices?
Last but not least, you can determine device-specific bugs
This is almost impossible with an emulator
If you do find a device-specific bug, the challenge is to fix it without breaking compatibility with other devices
Testing with mobile devices is (at least right now) a manual process
If your application is running slowly, you do not know whether it is the carrier's network,
your application, or a combination
Only by trial and error can you identify problems
Must ensure that mobile apps are working on all type of devices
(Smartphones, Tablets, etc.)
Compatability with major brands (various models from Samsung, Sony,
Nokia, HTC, Apple, etc.) and on all the platforms (iOS, Android,
Windows, etc.)
New devices are hitting the market so quickly that it may be
impossible to cover all the major devices
Complex User Interaction Scenarios
Desktop applications limit user interaction to using the mouse
More complex scenarios can occur while using mobile applications
Interactions through complex gestures, eye movement, direction, tilt,
movement, acceleration, GPS coordinates, surroundings, sound, etc.
Need to ensure that the app works as expected when the user interacts
with the app in different ways
Testing Resource Utilization of Mobile Applications
When developing a desktop application, an inefficient algorithm is not
necessarily a serious issue
Limited resources on mobile devices make it important to have
efficient algorithms
Good testing approaches include...
Testing for average cellular network/CPU/Memory/Thread usage
Excess usage of resources will limit the battery life of the mobile
device
Ideal testing strategies include testing common day-to-day
functionalities and measuring resource utilization
Testing with Emulators
Usually the most practical and cost-effective, and it even has some advantages
Emulators allow for inexpensive and quick functional testing of your application
Emulators run on computers with more resources, such as faster CPUs and more memory
Fast response times during testing enables you to complete tests more quickly
Most emulators employ high-level scripting languages
You can create consistent, automated tests, which are less error prone and quicker than manual testing
Automated scripting also allows for easier and faster regression testing
Do not rule out using emulators for as much of your testing as
possible
Cost-effective and efficient way to eliminate most of your bugs
Emulators are available for variety of systems -- Android, Windows
Phone, iPhone etc.