Today’s hyper-connected, next-generation interfaces—including bots such as Siri and Alexa, and voice-based systems—challenge traditional testing environments. They require new testing strategies and techniques.
To provide hyper-connectivity, systems need to support seamless handoffs among different points of interaction while providing relevant and contextual information quickly. (See Figure 1, below.) Such demands on IT systems require teams to successfully pair hardware capabilities with intelligent software technologies such as location intelligence, biometric sensing, and Bluetooth.
Figure 1. Multiple touchpoints for a hyper-connected application. Image from Infostretch.
Modern systems require new thinking about testing. Here's what to keep in mind.
Four challenges in mobile app testing
In mobile app testing, the major challenges fall into these categories:
- While simulating a simple mobile app is easy, the addition of sensors presents complexities for creating appropriate stimulation strategies and deciding on the right tools.
- Existing testing tools, though built for mobile testing, are built for function automation and so fall short with sensor-level automation.
- Hardware-level access for automation requires code-level instrumentation.
- Mobile apps require interruption testing simulations, such as when they are disconnected by phone calls, lost Internet connections, working offline, etc.
Example use cases
Here are some examples of why new testing techniques are needed for this new generation of apps, and how testing needs to be different from traditional methods.
Devices that use Bluetooth Low Energy (BLE) technology for connectivity create challenges in automated testing while syncing data, connecting, disconnecting, and determining actions. This is especially true if there is interference from other devices and if those devices are operating under different protocols. Such challenges demand more than traditional testing capabilities.
Apps that use mobile cameras to perform tasks—such as a scanning app that uses the mobile camera to take a picture of text and then extract text from the image—have their own requirements for testing algorithms or testing optical character-recognition capabilities.
Apps that use location metadata present different sets of challenges. An example would be Uber or any other taxi aggregator app. The challenges these apps present in automated testing involve creating simulations for different scenarios, map rendering, object identification, and simulating movement of users between various locations.
Apps that require the device location for date and time create challenges for traditional testing. This is especially true when you're trying to automate testing of a device's local date and time, as well as when trying to determine and trigger notifications for user actions.
Apps that take advantage of biometric authentication require different strategies and techniques for testing of such apps in fingerprint scanning, as well as to authenticate and trigger actions.
Solutions, strategies, and tools
Since mobile apps require testing many different features, ranging from Touch ID to GPS to gyroscope, they are difficult to test manually. Sometimes you must do testing individually, and sometimes you can do it for a bundled combination of apps. One way to mitigate such challenges is by using a mobile automation library.
Figure 2. A framework to automate testing of hyper connected apps. Image from Infostretch.
The mobile automation library can be part of your mobile application package; it doesn't require any code-level changes, and you can control it through the back-end interface. It can be used as part of regular automation, but also provides flexibility and configuration options. The goal is for you to be able to test a single mobile application for multiple scenarios, with the same package, without having to make any code-level changes.
Bot testing needs different approaches
Testing for bots represents a different set of challenges. Bots, be they voice or text, need to be tested for two different parameters:
- Common factors such as intent validation and response validation
- Specific factors for a particular bot
Automating the testing of bots thus creates a challenge to accommodate testing for both factors. And, while testing for common factors can be automated, testing for specific factors can be difficult.
Mitigate these challenges by adopting a layout approach for testing automation. This requires a framework layout that tests for commonalities between all bots with respect to intent and response.
By having a framework for commonalities testing, the testing effort for different bots would require just building specific automation for specific factors of the bot, and thus reduces the effort. This kind of separate framework for testing helps in modular automation, as well as in the scale of automation.
Automated bot testing should follow two approaches: intimating users' actions, and headless testing.
For intimating users' actions, select a framework that allows testing of the bot's end-to-end flow per test scenario by intimating users' actions and interacting with the bot. Such a framework would then capture the response of the bot and compare it with the desired response data.
For headless testing, your framework should spoof and directly connect to the bot server of the bot under test. Users' actions should then be sent directly, bypassing the bot channel and capturing the response of the bot, again comparing it with the desired response.
Testing for voice bots adds further complexities to the already challenging testing scenario. Voice bots require testing for a range of factors, such as accents, utterances, punctuations, background noise interferences, and distance from voice, in addition to the factors specific to chat-based bots.
Another testing frontier
While it can be said that application testing has seen some great innovation and technological advancements in recent years, the demands from advanced technologies have also become more complex. Today, testing not only demands technological advancements and strategic innovations, but it also requires a certain level of maturity and experience. Testing enterprises need to be experts in the field.
For more on how to automate testing for next-generation interfaces, including for bots, Alexa, and mobile, see my presentation, "How to Automate Testing for Next-Generation Interfaces (BOTs, Alexa, Mobile)," on October 3 at STARWEST. The conference, in Anaheim, California, runs from September 30 to October 5, 2018.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
