Foreward:
There
is a huge discrimination between Managing a traditional/regular web
testing project and Managing a Mobile Application Testing Project.
The group of challenges is/are unique to the mobile testing world,
and is/are quite different from traditional desktop or web delivery
testing models.
Today
the number of mobile devices, mobile operating systems, firmware
updates, and other possible customizations create an impossibly large
set of testing permutations, which comprises of a large diverse
range. With all of these testing permutations, the cost and
complexity of managing QA and testing in the mobile world is a very
large challenge for many organizations.
The
phenomenal growth of mobile devices has opened up avenues for a lot
of app development & a consequent need for testing the same. In
fact in certain write-ups & discussions the term apps-economy has
also come up.
Upcoming
mobile apps deliver complex functionality on platforms that at times
have limited resources for computing, though that constraint is being
overcome for many smartphone platforms.
Also,
unlike the PC, the mobile environment comprises a variety of devices
with diverse hardware and software configurations, device
capabilities and communication intricacies. This variety presents
unique challenges in development, quality assurance, and roll-out,
requiring unique testing strategies.
I
would like to share my views for managing a mobile application
testing, which is complex and dynamic in nature, project based, on my
knowledge and experience.
Pointers
for managing a mobile testing project:
- Need and Scope of the mobile app
- Deciding the Mobile Testing Strategy
- Targeted platform and domain of the app
- Working out with ideal test plans
- Identifying the devices for the targeted platform
- Setting up the test environment and testing process
- Estimation of the testing efforts and device logistics
- In accordance to the changes of guidelines in App Store submission/approval
- Constant mobile technology evolutions
- Identifying risk - mitigations
Need
and Scope of the mobile app
The
need and scope of the mobile app gives a basic understanding of the
viability, which initiates the actual thinking process and further
planning.
The
need of the mobile app gives an insight with regards to the core
objective/purpose, which, when clear would aid in understanding the
functionality of the app to a greater extent.
The
scope of the mobile app gives an insight with regards to the clear
scope of the app's functionality and the scope of the types of
testing to be done.
Also
based on the scope, the kind of test environment to be required to
test and if any support for automation tools would be needed, can be
derived at.
Deciding
the mobile testing strategy
Deciding
the ideal mobile testing strategy for the AUT plays a pivotal role.
Based on the time estimates and domain of the app, the corresponding
testing strategy should be chosen, to start with.
Specific
considerations for deciding a test strategy based on the domain of
the app needs to be taken care. Apart from traditional functional
testing, Adhoc testing plays a major role. Adhoc testing for mobile
app testing can be categorized into two types – adhoc testing on
the app and adhoc testing as per device specific. Few things should
be taken into consideration according to device specific
functionalities.
There
is a huge differentiation when testing a business app and when
testing a social networking app and when testing a gaming app and
when testing a finance domain app, the strategies vary accordingly.
The
thinking with regards to strategy has to be made/developed in such a
manner, that those strategies should be able to adjust to the diverse
and dynamic mobile platform.
Novel
strategies like testing within the company by different groups with
regards to the usability of the app, similar to beta customer testing
should be implemented. Testing the app in this manner gives the
scope of user experience and usability issues.
Targeted
platform and domain of the app
The
domain of the app along with the targeted platform plays a pivotal
role in managing the AUT. The targeted platform of the app gives the
scope of the target users in the market which needs to be taken into
consideration for the development/testing.
Every
platform has its own share of users in the market and this has to be
taken into consideration. Also the constant changes with regards to
the corresponding platform is also accountable for the consideration.
The huge complex diversity in mobile devices with regards to various
features, screen resolutions, memory considerations needs to be taken
into consideration.
Based
on the domain of the app - Social Networking, Finance, Banking,
Entertainment, Game, Health Care, Pharma etc., corresponding
precautionary measures along with the security concerns has to be
taken into consideration. For Social Networking and Entertainment
type of apps utmost care to has to be taken with regards to UI and
user-friendliness of the app; for Finance and Banking domains
security of the app needs to be taken into consideration.
The
test cases and types of testing may slightly vary based on the domain
of the app.
Working
with ideal test plans
Once
the testing strategy and testing life cycle is chosen, need to focus
on choosing the ideal test plan that suits the AUT.
The
test plan should cover what to test, when to test, how to test and
who to test. The list of devices along with OS needs to be listed
under the targeted devices and the time schedules along with the
release plan should be included in test plan.
The
exact scope (In Scope and Out Scope) should be clearly documented and
if the usage or need of any automation tools to be used.
Identifying
the devices for the targeted platform
Based
on the targeted platform of the app, the corresponding supported
devices needs to be identified so as to make sure that the app is
tested across a wider range of supported devices.
The
devices needs to identified based on
- multiple OS versions
- target users of app and domain specific
- locale considerations wrt usage of app
- app store/OEM statistics wrt to usage of apps
- nature of device or input type – touch, semi touch, standard, smart phone
- various Screen resolutions
- various technologies – pinch, zoom
- various OEMs
- network connectivity type
- in built hardware features/components
- devices with optimal hardware resources
by
taking all the above considerations, the devices needs to identified
and make sure almost all the devices are covered under testing for
the corresponding platform.
For
iOS platform, the device matrix is minimal, but for Android the scope
is a bit huge, and with the constant launch of new OEMs into market,
the device matrix becomes more complicated.
As
its practically not feasible to test on all physical devices,
simulators and automation tools which provide devices on cloud should
be taken into consideration.
Setting
up test environment and testing process
The
test environment and accordingly the test process has to be set up
once all the above mentioned factors are in place. The set up may
vary slightly according to the domain and platform of the app and
also varies if the app is of stand alone app or client-server based
app.
The
whole process involves the following:
- Setting up OTA process or Server
- If the application is client server based, then access to the server etc
- If the app is having scope of Database testing, then a separate test database environment
- If automation is required, then we should decide the better framework to use along with the tool
- Build delivery process should be defined
- Bug reporting process
- Summary reports for each Sprint.
- Defect Metrics
- Support for simulators
- Different types of devices – mobile, tablet,
- Compatibility considerations
Estimation
of the testing efforts and device logistics
Estimation
of the testing efforts and device logistics is a huge task to the
management team. The team has to make sure that the AUT has to be
tested over a broad and wide variety range of devices in the market
to make sure that the app supports all the OS versions along with the
features.
Its
not possibly feasible to test the app on all the available and
supported physical devices. The team has to come up with a device
matrix and make sure the matrix covers the overall variety and range
of supported OS versions and features.
And
also new devices along with new OS versions and features keep on
launching frequent which makes the task more cumbersome.
The
management has to keep this in view and go for the estimation of
device logistics and invariably testing efforts, as both are
interdependent. As mentioned above, its not possible feasible to test
on all physical devices, so need to look out for simulators, few
tools which provide device simulators like Nokia RDA, Samsung etc..
The
mobile market owing to its dynamic characteristics is extremely
volatile. Based on the value of the business, there has to be a
great thinking for the significant budget, if not this may lead to
issues in ROI.
Changes
of guidelines in App Store submission
Each
App Store has its own set of guidelines for a certain app to be
submitted in its corresponding app stores, which are generic and
independent of functionality of the app.
The
guidelines of App Stores keep on changing over the period of time in
adherence to the security impact or market trends and the team has to
be aware of the constant and frequent changes. If the team is not
aware of the changes in guidelines, chances of rejecting the app for
a silly reason may be high.
Constant
mobile technology evolutions
Mobile
technology is drastically and constantly changing with the evolution
of new technology changes. Frequent and constant changes with regards
to new OS versions, new features and new technology keep on
developing the mobile market industry. New OEMs keep on evolving,
almost everyone aiming at unique and special features to stand apart
in the market and to create a niche for their own branding.
The
team has to be aware of the constant mobile technology evolutions and
make sure they ramp up to the new challenges in the technology and at
the same time making sure the AUT supports the features to a greater
extent.
Its
not feasible to keep updated with the constant on-going technologies,
but needs to be done on a periodical basis and make a note of the
changes and update the skill set to match the same.
Identifying
risk – mitigations
- Supporting new os version – example iOS 8 beta version (at present)
- Supporting new and emerging technologies
- Schedule slippages
- Unavailability of devices
- Network outage issues
