What Mobile Testing Really Means

There’s been a lot of talk lately about Mobile First.
Here’s a starting point of how to think in terms of Mobile Testing.

More than two-thirds access the internet on a mobile device. Here’s a starting point of how to think in terms of Mobile Testing and make sure your site is ready for modern commerce needs.

Source: https://zoonou.com/resources/blog/mobile-testing-approach/

“When mobile comes to picture, the most important feature is user-friendliness. No matter, how good your application is, display-wise or feature-wise, if the user is not able to handle it on that little screen for more than 5 seconds, he will never come back.” — Why Mobile Testing is Tough

➡️Devices, Operating Systems, Screen size, Browsers, Versions, Extensions

Real devices

• Challenges: geolocation, battery drainage, sensors, SMS, calls

Pros: Highest accuracy, closest to real user experience.

Cons:

• Not as easy to choose which devices to use or buy, not cost-effective
• High # of Devices with different memory, network, and screen limitations
• Additional expenses for maintenance, limited access to a physical device
• Popular devices vs edge cases, different OS’s different screen resolutions
• Not all mfrs. of Android devices offer OS updates at the same time

Simulators

• Does NOT replicate hardware but can modify the environment.
• Pros: Easier setup than hardware, fast working, verify/explore behavior, cost-effective
• Cons: Device hardware not considered, false positives allow tests to pass, incomplete test

Cloud

• Pros: availability, multiple OS/networks, update apps, cost, scale, scripts run in parallel
• Cons: Loss of control, lower security, less control over internet connections

It is suggested that it would take 399 devices to be confident enough that your app works. 7 = 25%, 32 = 50%, 107 = 75%, 254 = 80%, 399 = 90%

➡️ Network Speeds Differ: 3G, 4G, Wifi, Broadband

➡️ Tools Suitable for Desktop testing are NOT suitable for Mobile

• There is no Swiss army knife.
• All tools are great in their own way, all tools suck in other ways.

➡️ Multiple Input Channels: Keyboard, Voice, Gestures, Pencil

➡️ Types: Mobile Web Apps, Native, Hybrid

• Mobile Web is Easiest to develop, update and requires no installation
• Native is difficult and expensive to develop, higher maintenance
• Hybrid (Native with embedded browser) is more cost-effective, easier to distribute changes, and can make use of some device features. It will not be as fast as native and cant make as much use of hardware.

➡️ Manual vs Automatic Testing

• Manual Pros: Cost Effective short term, flexible, user actions simulated better at the same pace Cons: Hard to reuse, less effective execution, slow, can’t load test or fully isolate tests.
• If the application contains new functionality, test it manually.
  If the application requires testing once or twice, do it manually.
• Automatic Pros: Suite tests are faster, cost-effective long term, automation reuse, only real option for performance testing Cons: Mobile testing tools are limited, time-consuming, less efficient in defining user-friendliness or positive customer experience
• Automate the scripts for regression test cases. If regression tests are repeated, automated testing is perfect for that. Automate the scripts for complex scenarios which are time-consuming if executed manually.

Testing Categories

➡️ Documentation

Prep stage. Layouts, Requirements, Completeness, Consistency. Contradictions must be resolved before beginning. Test Plan, Test Cases, Traceability.

➡️ Functional — Working as per requirements

Install, Run, Exit, Uninstall

• App type, target audience, distribution channels
• Installing, Running app, the device meets system requirements
• The app starts correctly, user manual available, operation, exits
• App permission testing (Camera may not operate if Storage is not allowed)

Fields

• Required fields work, operation fields operate regardless
• Verify displayed data corresponds to user interaction

Business Logic

• Buying, Shopping Cart, Support Payments Options
• Recovery of Purchase Regardless of Device
• Actual User Acceptance Testing (This is a large endeavor).

Interruptions

• Incoming calls/texts, SMS, MMS interruptions
• Battery discharge, removal, charging
• Disconnect wifi, add/remove SD card

User Feedback

• Downloading content, progress bar, button usage, reaction, error messages
• Destructive actions, synchronize sound, vibration, visual accessibility

Updates

• User data saved, progress saved/displayed before exiting by any means
• Updates to the application are supported via USB, wifi, 3G, Bluetooth

Device Resources

• Lack of space, Memory Leaks, Long running tasks, Power consumption
• How does it handle switching from 4G to 3G, removal of SD card, and interference with other apps?

More Resources

• Google Analytics, User Actions, Social Media Interactions

➡️ Usability: Satisfaction, Efficiency, Effectiveness

• Buttons, checkboxes, headings, icons, vector elements in correct size and placement
• Are touch elements too small and is it possible a finger thickness causes element overlap?
• Multitasking, Navigation of modules, ensure icons work
• Verify color scheme, text clarity in size and style, grammatically correct text
• Zoom in/out, context menus not overloaded
• App termination, resume state, synchronize user actions
• return/cancel actions, verify speed of response

➡️ UI design / UX

• Compliance with standards of UI
• 640x480, 800x600, 1024x768, 1280x800, 1366x768, 1400x900, 1680x1050
• Responsiveness of different devices and screens
• Multi-touch support versus forced single touch zones
• Elements: buttons, icons, colors, links, font sizes, layout, text boxes, formats, lists
• Advertisements don’t overlap, non-return screens, portrait/landscape
• Orientation: Does the application work in both portrait and landscape?

➡️ Compatibility

• OS: Windows, iOS, Android, Blackberry
• Browser: Firefox, Chrome, Opera, Safari
• Database: Oracle, DB2, MySQL, Sybase, Local Storage
• Device: Phone, Tablet, RAM, Processor, Resolution, Battery
• Networks: 2G, 3G, 4G, wifi, broadband

➡️ Performance

• Operability, Stability, Resource Consumption
• Response time under normal versus heavy load
• Capacity of application under conditions exceeding stress levels
• Long time workers under volume testing
• Determine. the number of concurrent users supported
• Run the same under different network conditions
• App and Infrastructure bottlenecks, How do we cope with load volumes?
• Response time meets requirements
• Application Stability Under Hard Load
• Performance Under non-permanent connection to the internet
• Identify Performance Bottlenecks
• Verify Application’s readiness to go live

➡️ Security

• Logins, passwords, payment info, caching, cookies, auth tokens with SSL
• Protect the backend system against SQL injection
• Protect against Denial of Service attacks and malicious users
• Protect the application itself from surrounding applications running on the device
• Interaction of system files, personal info stored in device
• How does the application handle memory leaks and overflow bugs if they exist
• Threat analysis and modeling, Vulnerability analysis, Rooted & Jailbroken devices
• Protect against storing logs with confidential information
• Malicious third party code

➡️ Recovery

• Verify effective recovery after crash, break in connection, back end system failures, transaction purchase failures, low battery, out of storage, reboots
• Clearing data cache should log out user and not preserve personal info

➡️ Localization

• Languages supported
• Correct translations (both individual words and grammatically sensible)
• Graphics might contain text or images that require modification
• Date formats, Numeric delimiters

➡️ Change Related Testing

• Verify all known bugs have been addressed (confirmation)
• Verify that new changes do not cause new bugs (regression)

➡️ Field Testing

• Get as close as possible to the end-user environments (will it work inside? on the range?)
• Network, WIFI routers, Hardware out of date at locations where iPad app may be used
• Use application as the passenger in a car in order to test affect of unstable network on app
• Are you testing to verify the app against requirements, or to get the full end-user experience?
• Gives a team great advantages in knowing how the app will perform at the desired locations.
• Verify that the app is stable as every new version is released as soon as possible
• Log in as real user, obtain permissions. Change locations, sometimes test in transit.
• Verify different networks, perform transactional / cart based actions & break connection.
• Part of the testing cycle and should never be avoided or taken lightly

➡️ Certification

• Google: apk program policies, UIG requirements, no viruses (lose ability to rollout)
• Apple: Human Interface Guidelines, Unique Name, Verified signatures, Prohibited APIs

Learn More: Mobile App Testing Tutorials (A Complete Guide With 30+ Tutorials) https://www.softwaretestinghelp.com/beginners-guide-to-mobile-application-testing/

Now with 20 years of experience, Dan Emmons strives to make issues that seem complex, overwhelming, or insurmountable more manageable for the Team, and to provide exceptional service that exceeds clients’ expectations. For more information, contact him on LinkedIn or through Emmonspired LLC.