When it comes to QA, I am often asked, "What should my testing strategy be?" The answer, unfortunately, isn't clear-cut because it depends on your product's level of maturity.
Think back to the early 2000s. Social media were a new and upcoming phenomenon, and websites such as MySpace and Bebo were in no way yet mainstream. Early adopters were mostly young and willing to look past a bug or two, understanding that social media represented a developing and innovative platform.
But in the years that followed, the demographics of users changed.
It soon became apparent that social media were firmly in the mainstream. The late adopters began to include older people (much to the dismay of younger users!), with parents and grandparents signing up to sites such as Facebook and Twitter.
As the number of people using these platforms increased, the demographics of users changed. Companies had to adapt both their product and view of QA to suit their new user base. With this shift, the way companies looked at testing had to fundamentally change.
Here's why product maturity needs to be front-of-mind when developing your own testing strategy.
The 3 stages of product maturity
When writing the book Leading Quality with Owais Peer, co-founder of Global App Testing, we discovered that product evolutions like these were extremely common. After conducting interviews with over 120 quality experts from companies including King, HelloFresh, GitHub, Reddit, and eBay, we found that our clients' product evolution tended to move through three main categories.
Here's an explanation of each of those stages, and how you can use software testing to guarantee quality at every level of your product.
Source: Global App Testing
The validation stage: Flexible testing
The early iterations of Bebo and MySpace are great examples of products in the validation stage. Users at this stage are innovative, enthusiastic, and forgiving of software hiccups as the product rapidly evolves.
Developers focus on creating a minimal viable product and building an app that customers actually want. Software changes are frequent, as the product develops through challenges and opportunities, until the company either finds a promising prototype or abandons the idea altogether.
The testing method you should use
The core focuses of this stage are cost-effectiveness and flexibility. Your company is scrambling to find a product-market fit, and so your QA approach should only be testing core user flows and unblocking critical user issues.
For this reason, ad hoc testing is the most effective method to use. With this technique, tests can be conducted randomly, without any formal planning or procedure. This gives you the freedom to update your strategy as your product develops.
Automation is in no way a primary concern at this stage. Manual testing methods are the best way forward, since they allow more flexibility than the rigidity of automation. Manual unit testing, for example, could be crucial to this QA strategy, due to the fact that you can test each separate component of a product, validating each unit of software.
The predictability stage: Software stability
Once you have found your product-market fit, your priorities change. The focus moves from building an app fit for the market to making sure its core features are working effectively. You are aiming for predictability.
At this point, your product is beginning to grow in the number of users, perhaps including some demographics consisting of people who may be less forgiving of software bugs.
A study by Walker Information, a strategic-planning consultancy, stated that by 2020 customer experience will overtake price and product as a key brand differentiator. This shows that, for your brand to stand out at this stage, your customers need to have a great experience using your app. Therefore, predictable, stable software is hugely important.
The testing method you should use
Your QA strategy should include a blend of exploratory testing and automation to ensure full testing coverage. When King, maker of the global game phenomenon Candy Crush, reaches the predictability stage in one of its games, it implements a mixture of different tools and strategies.
It increases the amount of test automation, and uses a crowd of testers, both internally and through an external testing partner that uses crowdsourcing. This enables the company to discover and fix software issues that occurred during the previous (predictability) stage, when a team may have taken shortcuts in a rush to find product-market fit.
Over time, you will be able to prioritize which tests should be automated and which should use crowdsourced testing, but the main focus at this stage is utilizing this vital mix of exploratory and automated testing methods.
This way, you can discover bugs your developers may not have considered before, and fix issues that would become an even more significant problem at the next stage of your product.
The scaling stage: Minimizing negative impact and accelerating growth
At the scaling stage, companies focus on accelerating growth through new user adoption and increased engagement for existing users. The main QA concern at this stage is that even a small problem can impact a vast number of people.
In the previous stage validation, a bug that affects 1% of users isn't even on the quality team's radar. In the predictability stage, a bug like this is, similarly, a low priority. But at the scaling stage, 1% of your users could comprise more people than your entire user base in the first two stages combined.
For example, a bug that affects just a small percentage of Google Maps users could lead to over 10 million people being affected. Further, the more your business is intertwined with an application, the higher the stakes.
Tech-first organizations such as Uber are entirely reliant on their apps. A bug affecting Uber's UK users for just an hour could result in immediate financial losses.
The testing method you should use
At this stage, software testing is of the utmost importance. An efficient, well-planned, and extensive QA strategy is critical to ensuring that a bug does not affect even a small percentage of your user base.
Because of this, companies usually invest in a wide range of testing types. For example, King uses this period to delve deeper into performance testing, looking at battery, network, and CPU consumption to ensure the app is genuinely meeting and exceeding customer expectations.
Again, automated testing can also prove extremely useful in this scenario. Your developers know that your app is now mostly predictable, but do they know that it will be predictable when running thousands of test cases at once?
Finally, crowd-based localization testing can prove extremely beneficial to help drive growth. If you are looking to expand to a global user base, you need to ensure your product works effectively in countries all around the world. Focusing part of your testing strategy on localization will prevent any mistakes that could be a bottleneck to your company's global growth.
Effective QA strategy is crucial
As your product changes, your QA strategy should change with it.
There may not be one set method of testing for each stage of your product's evolution, but there are common challenges all companies need to overcome. Using a blend of different testing types that match the stage of your product's maturity will help you continue to scale.
Adopting a context-specific approach to testing, and continuously adapting your strategy, is the real answer to the question, "What should my testing strategy be?"
This article expands on "How Quality Changes with Product Maturity," Chapter 5 of the book Leading Quality, co-written by Ronald Cummings-John and Owais Peer and supported by Global App Testing.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
