What you should know about mobile Quality Assurance

First of all, we need to be clear that mobile testing is not fundamentally different than desktop or web based ones. If you are used to a specific methodology in your everyday testing, the chance is quite high that you will be able to use them on your mobile tests too. Of course, there are differences in functional tests, but the most obvious ones are in the performance and non-functional aspects. We should also keep in mind that even though new OS and their variations show up every day the general idea is for standardization and simplification of the code and tests as this can lead to easier transfer between platforms.

With that being said there are tons of differences and obstacles a QA engineer will face when testing on mobile devices (a good tester shouldn’t face any major issues as one of the main qualities he has to possess is the ability to adapt):

  1. The Aspect Ratio And Screen Rotation
    First big difference is regarding the aspect ratio as mobile devices (tablets included) usually have very small screens compared to PCs and Notebooks. Because of this, the testing of User Interfaces becomes essentially harder. Even with a big variety of simulators, one can face the issue that the required resolution won’t be available or that the emulator won’t show the simulation properly. 
 If a developer or QA engineer wants to create simulations for Microsoft Surface 3 for example, one huge problem will come on top – there are almost no emulators for that specific device. Of course, there is the possibility to simulate it through “Visual Studio” with added plug-in, but once the person does that and use it he will be quite unpleasantly surprised because the plug-in doesn’t have anything in common with the real resolution of the device.
    Another issue that desktop and web developers and QA engineers don’t face is the rotation of the screen, supported by most mobile devices. Many customers or Product Owners require a rotation to be available in their applications, but this can cause additional trouble when it comes to testing.
    Another problem is the vast amount of tests a mobile application has to go through – as the size of phone displays can vary significantly even when it comes to the same manufacturer. For example, the iPhone 5 has a 4” display, whereas the iPhone 6 is 4.7” on the diagonal. When you add in the iPhone 6 Plus (5.5”), iPad Mini (7.9”), and iPad standard (9.7”), it becomes harder and harder to code and test mobile applications that look “good” on all screens.
  2. The second big difference can come from the limitations of storage and processing power of the mobile devices. Even nowadays when many phones have quite good capacities they still are no real rival of the capacities of even the ordinary desktop PC. This by itself is a good enough reason for mobile applications to require much more performance testing than the desktop and web based programs.
  3. Third on the list is the huge variety of mobile OS platforms that are available on the market. First the user can choose from Android, iOS, Windows mobile, Blackberry OS and some other, not so popular options. Then you face the different modifications for each one – only for Android there are hundreds that can be found over the net. Each one has its own specifics.
    For example Samsung uses its own version of Android, which is much tweaked compared to the original. Then you have One plus phones which use extremely modified OS, based on Android.
    This diversity in the software can cause one mobile application, that runs perfectly on Nexus and Samsung devices to crash on LG manufactured ones and to look very distorted on Sony ones for example. Then there come the updates. One can have perfectly running app build for Android 5.0, but after the upgrade of the OS to 5.1 the same app can stop working. This can be caused by serious changes in processes or something as simple as a change of permissions.
  4. The fourth difference appears when it comes to location and internet connectivity. There are 3 options for Desktop apps – wi-fi, cable or no connection. One has different and more options for mobile apps – 2/3/4G networks, wi fi, tethering, no connectivity, active or inactive GPS and all of them can influence the performance of the application. In addition mobile QA engineer should also check if software permissions are covered and implemented properly, all permissions are called with installation or in the worst case when specific functionality is called.
  5. Fifth on the list is difference in the accessibility. A desktop user is used to interact with applications mostly through mouse and keyboard. And even in the cases when audio support is implemented it heavily relies in tags in the code. For comparison in the context of mobile device and application there are much more ways for interaction. In some older models smartphones have both keyboard and touchpad available, but even then one of the biggest improvements was the introduction of touchscreen. Nowadays the user is not only presented with keyboard and touchscreen options but also with gesture, fingerprint, eyesight and voice control. The mobile manufacturers and developers are aiming to make the use of the device as personal and intuitive as possible, but this causes additional work for the testers as they have to figure out all possible actions and movements the user can make so no bug is missed.
    With so many sensors and patterns it’s not unusual for malfunctions to appear, which can additionally prolong the test process. In those cases the QA engineer have to figure out if a new bug has appeared or the device caused the issue. One of the best approaches in case you’re not sure if you found a bug or device malfunction is to restart the device and see if you can reproduce the issue. If the issue is reproducible, but you still are not sure if it’s a bug open another application that uses the same functionality. If you face the same issue then probably you came across a device malfunction. If it’s not reproducible then it’s a good idea to contact the developer as you found a bug.
  6. The sixth one is frequency of changes, rapid installations and removals. Even though web and desktop applications also get updates, the frequency with which the mobile applications, the OS and the permissions are getting improved or changed is tremendous. If one web browser is getting updates every couple of months and the desktop OS yearly, the average update of an Android app is once or twice a month, and the OS updates 3-4 times a year.Such intensive changes can cause malfunctions in apps that used to be working normally prior the update. This means that at least smoke test is required to be performed after each new build and version.
    Other factors include installation and removal of the applications itself. The mobile QA engineer should check if new updates can be installed over old builds without issues and loss of data. The tester should also observe the installation process as in many cases there are version specific libraries used by the developers and this can cause malfunction during installation. Once the previous test is performed the QA engineer should check if the app can be removed safely, doesn’t cause performance issues after removal and the product can be installed without issues again on the same device.
  7. Security and privacy concerns are the next ones on the list. Even though all mobile OS developers aim to make their system as secure and stable as possible there is still huge difference in the way how they behave compared to their desktop counterparts. Almost all applications require interaction with first class security features, like camera, phone contacts, google account, gallery and etc. These are all personal user data, and any defect around the (unintentional) misuse of this data can jeopardize trust of the application.
    In mobile world, it is important to ensure that applications are secure from the intruders, and it is equally important to ensure that applications are not intruding or accessing data unintentionally. What developers and QA engineers should not forget is that users wouldn’t install an application which requires too many permissions, especially if it requires their personal data. Exactly because of that the mobile testers should check them and if some can be dropped contact the programmers to remove them or figure a workaround. For example, it’s better to redirect to Gallery app instead to ask for permission to open it in-app.
  8. Eighth in the list is cross platform testing. There are substantial differences in interaction between user and app on different platforms. The QA engineer should always expect different behavior in apps that are available for more than one platform. This is caused by difference in permissions, access, design and system requirements. One can be an expert tester when it comes to Android, but if he has never worked on iOS he might find himself lost in all those differences. So for mobile QA engineers, it’s important to know the fundamental differences in the platforms. 

Perfect example for this is a situation I faced when I had to test a social network for sharing images. We had all the requirements covered for the Android app, it was working quite well and we published it. In the meantime, we were going for a release on iOS too. That’s when we faced quite unexpected block – we were blocked by Apple with a requirement that at least some of the features should be available without registration. This caused us to change both the Android and the iOS versions so we can comply with the rules of Apple store.

When talking about mobile testing we should always keep in mind that mobile devices don’t behave the same way as desktop computers and even laptops. That leads to a huge difference in the way apps work. Because of the smaller screens, processing power and the reliability of battery any mobile device can’t multitask in the same sense as the other ones. First, the user can’t really use more than two apps simultaneously and that’s only true in cases of some flagman phones otherwise, the limit is one active responsive application (This excludes some of the audio apps, yet they still can be affected by this). Because of that, we should always consider how the app will behave if it’s not active but is not closed (active in the recent apps tray).

First, all apps have a priority status, this means some apps will start with the device and can’t be killed. Usually, those are the system apps, but in some cases, such behavior can be expected from other apps too.

Second, when an app is sent to recent apps screen (either other one is opened on top or home screen button is pressed) it can stop behaving like an active one to save battery for example. In these cases when reopened it might need to refresh the data that will be displayed or to restart, which means that if there is any unsaved data left it will be lost. Such refresh requests are usually embedded in the code and need to be tested.

Third, QA engineer should also consider the behavior of the app when the battery is low or the user has turned on either power or ultra-power saving modes. Each of the three can interrupt the normal processes of an application. For example, if we have to test application which is running system processes, but is not with priority status the OS will automatically kill it so it can save battery. 
When it comes to power saving mode some applications might not be available, especially those with high consumption of energy. Such example is the camera app. In some cases, if too much processing power is used we can observe lag and crashes in the application. In those cases, we should try to figure out workarounds so less power is used and higher stability is achieved. 
When it comes to ultra-power saving mode we face even further difficulties as only specific apps are allowed to work under this regime. This might require for the QA engineer to manually add the app that’s being tested to the list.
One of the most important differences comes with the mobile specific functional and non-functional testing. Even though I started the article explaining how there are no fundamental differences when it comes to methodologies, there are some mobile specific tests one should keep in mind when testing.

The most functional tests that would be new for the QA engineer are the ones connected to positioning, gesture control, and phone services. But the more challenging ones are the ones that are relying on system permissions. In those cases, the tester should be sure he is very familiar with the specifics of the OS and the device as basically any functional issue can be a result of forgotten permission and each wrong assumption can lead to prolonged development time and delays in delivery. 
Another aspect of the mobile functional testing is the notifications. Even though recently notifications became available for the desktop OS platforms like Windows 8 and 10 it’s still rare see them and they are quite limited compared to their counterparts on mobile devices. Only this aspect of testing is very time-consuming as there are many possibilities for a notification to be triggered. Also, there are different types of notifications – pop-ups, small and big notifications in the notification bar. Each of them requires a specific set of permissions and might not be available for different versions of the OS.

The majority of new tests, that need to be performed are non-functional though. The QA engineer should check things like how much data is the app using, or how expensive it would be for the user to use the product, how much battery is the app consuming? How does the app behave if there are many background processes running, how it behaves when battery is full or empty? How much space is needed for the installed app, how much space is needed for additional resources connected to the app? What trail is it leaving and how well it removes such trails?
In the end we have multiple technologies available for mobile devices. We can have mobile versions of web sites and apps, and then there are the native apps that are built and available only for the mobile OS. And in the end we have the hybrid apps, which incorporate both types.

When it comes to the mobile versions of HTML based sites and apps there isn’t any huge difference with their desktop counterparts. They are usually simplified when it comes to design and follow a standard format. Most issues come from the design as the developers have to make the mobile version as diverse as possible so it can be available on most devices.

Native apps are more complex and allow more freedom for the user. However, because of their complexity, they are more prone to failures and bugs and require much more time for testing. They are usually the ones that require special sets of system permissions and mobile specific tests.
Hybrid apps are mixture of both previous types of mobile software and as such incorporate specifics from them. Even though in most cases they are simpler than native apps they can fail in unexpected situations, the HTML can conflict with the native one, or frameworks can fail to load and display the required information. Design issues are also common and because of the frameworks, they might be harder to fix. Another aspect of the hybrid apps is that they need internet connectivity to work properly and when such is absent either cached version should be available or user should be informed some functions won’t be available because of the lack of connection.

Mobile QA engineers usually have an equal time or even less to test their products compared to their colleagues who test desktop applications. Yet they face more testing because of the already listed differences between the two platforms. The good test planning is fundamental for mobile applications otherwise there is huge chance for important tests to be missed, which can cause issues in later plan. The mobile QAs should work closely with everyone in the project from the start so proper planning is created. They also need to be familiar with the tendencies and user’s thinking. The positive side is they usually can start testing in earlier stages of development and because of that win some precious time for testing. And finally, the good QA engineer should not face any other problems than getting to know the specifics and test environment when switching between the two platforms. The adaptability and the sharp eye for the detail are the two most important qualities of the software tester.

Leave a Reply

Your email address will not be published. Required fields are marked *