Managing a Mobile Application Testing Project

Foreward:

There is a huge discrimination between Managing a traditional/regular web testing project and Managing a Mobile Application Testing Project. The group of challenges is/are unique to the mobile testing world, and is/are quite different from traditional desktop or web delivery testing models.

Today the number of mobile devices, mobile operating systems, firmware updates, and other possible customizations create an impossibly large set of testing permutations, which comprises of a large diverse range. With all of these testing permutations, the cost and complexity of managing QA and testing in the mobile world is a very large challenge for many organizations.

The phenomenal growth of mobile devices has opened up avenues for a lot of app development & a consequent need for testing the same. In fact in certain write-ups & discussions the term apps-economy has also come up.

Upcoming mobile apps deliver complex functionality on platforms that at times have limited resources for computing, though that constraint is being overcome for many smartphone platforms.

Also, unlike the PC, the mobile environment comprises a variety of devices with diverse hardware and software configurations, device capabilities and communication intricacies. This variety presents unique challenges in development, quality assurance, and roll-out, requiring unique testing strategies.

I would like to share my views for managing a mobile application testing, which is complex and dynamic in nature, project based, on my knowledge and experience.

Pointers for managing a mobile testing project:

1. Need and Scope of the mobile app
2. Deciding the Mobile Testing Strategy
3. Targeted platform and domain of the app
4. Working out with ideal test plans
5. Identifying the devices for the targeted platform
6. Setting up the test environment and testing process
7. Estimation of the testing efforts and device logistics
8. In accordance to the changes of guidelines in App Store submission/approval
9. Constant mobile technology evolutions
10. Identifying risk - mitigations

Need and Scope of the mobile app

The need and scope of the mobile app gives a basic understanding of the viability, which initiates the actual thinking process and further planning.

The need of the mobile app gives an insight with regards to the core objective/purpose, which, when clear would aid in understanding the functionality of the app to a greater extent.

The scope of the mobile app gives an insight with regards to the clear scope of the app's functionality and the scope of the types of testing to be done.

Also based on the scope, the kind of test environment to be required to test and if any support for automation tools would be needed, can be derived at.

Deciding the mobile testing strategy

Deciding the ideal mobile testing strategy for the AUT plays a pivotal role. Based on the time estimates and domain of the app, the corresponding testing strategy should be chosen, to start with.

Specific considerations for deciding a test strategy based on the domain of the app needs to be taken care. Apart from traditional functional testing, Adhoc testing plays a major role. Adhoc testing for mobile app testing can be categorized into two types – adhoc testing on the app and adhoc testing as per device specific. Few things should be taken into consideration according to device specific functionalities.

There is a huge differentiation when testing a business app and when testing a social networking app and when testing a gaming app and when testing a finance domain app, the strategies vary accordingly.

The thinking with regards to strategy has to be made/developed in such a manner, that those strategies should be able to adjust to the diverse and dynamic mobile platform.

Novel strategies like testing within the company by different groups with regards to the usability of the app, similar to beta customer testing should be implemented. Testing the app in this manner gives the scope of user experience and usability issues.

Targeted platform and domain of the app

The domain of the app along with the targeted platform plays a pivotal role in managing the AUT. The targeted platform of the app gives the scope of the target users in the market which needs to be taken into consideration for the development/testing.

Every platform has its own share of users in the market and this has to be taken into consideration. Also the constant changes with regards to the corresponding platform is also accountable for the consideration. The huge complex diversity in mobile devices with regards to various features, screen resolutions, memory considerations needs to be taken into consideration.

Based on the domain of the app - Social Networking, Finance, Banking, Entertainment, Game, Health Care, Pharma etc., corresponding precautionary measures along with the security concerns has to be taken into consideration. For Social Networking and Entertainment type of apps utmost care to has to be taken with regards to UI and user-friendliness of the app; for Finance and Banking domains security of the app needs to be taken into consideration.

The test cases and types of testing may slightly vary based on the domain of the app.

Working with ideal test plans

Once the testing strategy and testing life cycle is chosen, need to focus on choosing the ideal test plan that suits the AUT.

The test plan should cover what to test, when to test, how to test and who to test. The list of devices along with OS needs to be listed under the targeted devices and the time schedules along with the release plan should be included in test plan.

The exact scope (In Scope and Out Scope) should be clearly documented and if the usage or need of any automation tools to be used.

Identifying the devices for the targeted platform

Based on the targeted platform of the app, the corresponding supported devices needs to be identified so as to make sure that the app is tested across a wider range of supported devices.

The devices needs to identified based on

• multiple OS versions
• target users of app and domain specific
• locale considerations wrt usage of app
• app store/OEM statistics wrt to usage of apps
• nature of device or input type – touch, semi touch, standard, smart phone
• various Screen resolutions
• various technologies – pinch, zoom
• various OEMs
• network connectivity type
• in built hardware features/components
• devices with optimal hardware resources

by taking all the above considerations, the devices needs to identified and make sure almost all the devices are covered under testing for the corresponding platform.

For iOS platform, the device matrix is minimal, but for Android the scope is a bit huge, and with the constant launch of new OEMs into market, the device matrix becomes more complicated.

As its practically not feasible to test on all physical devices, simulators and automation tools which provide devices on cloud should be taken into consideration.

Setting up test environment and testing process

The test environment and accordingly the test process has to be set up once all the above mentioned factors are in place. The set up may vary slightly according to the domain and platform of the app and also varies if the app is of stand alone app or client-server based app.

The whole process involves the following:

• Setting up OTA process or Server
• If the application is client server based, then access to the server etc
• If the app is having scope of Database testing, then a separate test database environment
• If automation is required, then we should decide the better framework to use along with the tool
• Build delivery process should be defined
• Bug reporting process
• Summary reports for each Sprint.
• Defect Metrics
• Support for simulators
• Different types of devices – mobile, tablet,
• Compatibility considerations

Estimation of the testing efforts and device logistics

Estimation of the testing efforts and device logistics is a huge task to the management team. The team has to make sure that the AUT has to be tested over a broad and wide variety range of devices in the market to make sure that the app supports all the OS versions along with the features.

Its not possibly feasible to test the app on all the available and supported physical devices. The team has to come up with a device matrix and make sure the matrix covers the overall variety and range of supported OS versions and features.

And also new devices along with new OS versions and features keep on launching frequent which makes the task more cumbersome.

The management has to keep this in view and go for the estimation of device logistics and invariably testing efforts, as both are interdependent. As mentioned above, its not possible feasible to test on all physical devices, so need to look out for simulators, few tools which provide device simulators like Nokia RDA, Samsung etc..

The mobile market owing to its dynamic characteristics is extremely volatile. Based on the value of the business, there has to be a great thinking for the significant budget, if not this may lead to issues in ROI.

Changes of guidelines in App Store submission

Each App Store has its own set of guidelines for a certain app to be submitted in its corresponding app stores, which are generic and independent of functionality of the app.

The guidelines of App Stores keep on changing over the period of time in adherence to the security impact or market trends and the team has to be aware of the constant and frequent changes. If the team is not aware of the changes in guidelines, chances of rejecting the app for a silly reason may be high.

Constant mobile technology evolutions

Mobile technology is drastically and constantly changing with the evolution of new technology changes. Frequent and constant changes with regards to new OS versions, new features and new technology keep on developing the mobile market industry. New OEMs keep on evolving, almost everyone aiming at unique and special features to stand apart in the market and to create a niche for their own branding.

The team has to be aware of the constant mobile technology evolutions and make sure they ramp up to the new challenges in the technology and at the same time making sure the AUT supports the features to a greater extent.

Its not feasible to keep updated with the constant on-going technologies, but needs to be done on a periodical basis and make a note of the changes and update the skill set to match the same.

Identifying risk – mitigations

• Supporting new os version – example iOS 8 beta version (at present)
• Supporting new and emerging technologies
• Schedule slippages
• Unavailability of devices
• Network outage issues