App testing: It’s not just for emulators anymore

Transparent gear shapes

“For every 1000 bad mobile experiences, one user will post a bad review in the App Store the rest will just delete the app.”

Let’s talk about a topic most companies would rather overlook… app testing. Too often organizations say to me, “we did some emulator testing on the original application, but we let our App Store reviews be our barometer of how the app is performing in the wild.”

“For every 1000 bad mobile experiences, one user will post a bad review in the App Store the rest will just delete the app.”

Of course this is the business equivalent of closing the barn door after the horses have already escaped. By the time the reviews decline, your users have already deleted your app.

And … like the horses … they’re not coming back.

Whether the user is an employee, customer, or partner, the most frequent interface that they will have with your brand or company is going to be that mobile application. This alone should underscore the importance of a highly functional well-tested application. Additionally, a poorly tested application will cost more to support, no matter the audience. Many companies understand how to use the emulator for testing, included with the native SDK, but very few utilize other methodologies of testing.

There are three different types of testing an app should undergo before it’s deployed to the public: emulator testing, on device testing, and, live network and performance/source testing. If you’re making minor modifications to the application for a re-release (less then 10-15% code changes), all three are probably not required.

Emulator Testing

Emulator testing is the best way to get a quick functional test of the application during the development process. After the application has been compiled, it’s then run through the emulator, which is a generic representation of all of the operation system specific functions available to an end-user. The emulator is excellent at testing logic application flow in general appearance. However, it cannot test information source, network, or true device capabilities. This is the stage that you should spend the least amount of time on, particularly for Android apps due to the massive OS fragmentation.

On Device Testing

On device testing or live device testing tests the performance of the application on a live connected device. That device could be connected to a Wi-Fi network or other network source, but primarily you are looking for the performance of the application on the actual device. You are looking at how variations in display, processor, speed, memory footprint, and other items may affect the applications performance.

Additionally, when there are specific versions of an OS (i.e. Verizon’s, Droid RAZR vs AT&Ts HTC inspire), an application may behave completely differently. Ideally you want the application to perform similarly across a wide audience of devices. The companies that offer live device testing will also produce testing scripts for you to use when testing applications. Often these test scripts can be used to generate support scripts and identify where network and device issues appear.

Live Network and Performance/Source Testing

Performance and live source testing enables you to test on a live network, not simulated, and gauge the performance of the information sources required for the application to function. For applications that make network calls this sort of testing is a requirement, however for those that do not have a connected element (fewer and fewer apps these days), this step is unnecessary. The reason behind this type of testing is that wireless network performance differs greatly from wired network performance. This is really a two way testing street, you not only want to test how the application will perform, but you also want to see how those applications will affect the performance of those information sources that will be impacted by the mobile applications.

What do you think, have I missed any pertinent testing requirements, or overstated the issues? How do you test, are there specific tools you use? Are there testing issues that show up over and over again? Appreciate your input

And a special thanks to the (amazing) Appcelerator QA team who provided a ton of great input for this post.

by Michael King


  1. Certainly, one of the perceptions that I have held for a long time … whether it is grounded in truth or not (perception is reality) … is this: Apple’s own iPhone apps are bulletproof; Android phone apps are unlikely to work correctly once you step out of their singular “comfort zone.”  
    This is literally an expectation on my part that these applications, in addition to being inconsistent one to the other, can be expected not to work and not to have been properly tested. Furthermore, that bugs if reported probably won’t be fixed (except by the spare-time developer who wrote it, in his spare time, when he can spare the time, maybe). 
    That’s a “pretty darned awful” feeling to come away with, but I don’t come by it except through painful and repeated experience. And the root cause of the problem certainly seems to be that the testing simply never occurred. At all. 
    I don’t want to download an application to “see if” it will actually work.