What's the best load testing strategy—to invest in a realistic test or take a quick-and-dirty approach?
Many testers aim for realism, but setting up a realistic simulation can take a lot of time and effort. This can significantly delay testing, leading to serious risks. As Kent Beck and Cynthia Andres observe in Extreme Programming Explained, catching problems early costs less than fixing them at the end of the development lifecycle.
The other option is to use "good enough" testing as early as possible. I would argue that in many cases, this approach produces better outcomes. We can spend 20 percent of our typical effort on the test configuration and still learn 80 percent of what we want to know—and we can find problems when they're still cheap and easy to fix.
The 80/20 rule for performance testing
The 80/20 rule, also known as the Pareto Principle, states that for many things, 80 percent of the effects derive from 20 percent of the causes.
Scripting a load test can be a great example of the Pareto Principle in play. Configuring a load test is difficult. There are tons of parameters you can tweak in order to achieve the kind of user behavior and simulation you want.
Many people approach load testing with the ambition to simulate reality perfectly in their load test, so they'll learn exactly how many real users their production system will be able to handle.
Unfortunately, simulating reality is difficult, which means it takes a lot of time and effort to set up such a test, and many times even a major effort will fall short and miss some critical aspect that causes the simulation to be unrealistic.
The end result can be a hugely expensive and/or time-consuming load test that might not even give you the results you want. Organizations that spend tons of time and money on a load test are not likely to do any load testing except when it's necessary.
Often, this will be just before a launch, when there's little time to fix any problems found and the cost of fixing them will be high because the problems are found at such a late stage.
Simplify testing and start early
Here's where the Pareto Principle enters the scene: if you don't try to simulate reality perfectly, your configuration will be a lot simpler. You can get away with spending only 20 percent of the effort of setting up the load test.
The result you get might not be a number telling you exactly how many users your site could handle in production, but it's likely to tell you if your performance is way off target, and it'll likely highlight your performance bottlenecks.
Because the test is much simpler to configure and run, you can run more tests at various stages of development and learn a lot more in aggregate than you would've learned from a single test.
Being able to run more tests earlier on makes it possible to detect and fix performance issues earlier, and fixing an issue as early as possible usually means saving development time, which drives down total project cost. More frequent testing will also result in a team that is generally more confident and more involved in product performance.
Best practices for load testing
Those were a lot of abstract, fluffy ideas, but how do you actually do it?
Exactly where to cut away complexity in your load testing will depend on your situation, but it could mean that you might want to use a simplistic version of user behavior when designing your load test.
The simulated user in your test might be a lot more static in its behavior than a real user, perhaps just accessing a set sequence of pages or content with no randomization. Or you may have your simulated users access a randomly selected page on your site, choosing to ignore the fact that some pages are much more heavily visited than others.
If you're testing an API-driven application (like a mobile app), you might find it better not to care about user flows and instead try to exercise API endpoints—either as a mix or perhaps even individually.
If you know how many calls per second API endpoint "X" will have at 1,000 concurrent users, you can run a load test that generates that number of requests per second and see if your back end can handle it and where in your system the bottlenecks appear. Then do the same thing for each endpoint (you can of course exercise multiple endpoints in the same test if you want).
Often, you won't get a lot of additional information from properly simulating user flows, and it complicates the load test configuration substantially. The absolute simplest test you can do is just hammering a single API endpoint at a time. This is easy to configure, but if you run that type of test repeatedly for each important endpoint and from an early stage in the development of your application, you'll be confident about your performance come release date.
Build up to realistic simulation
There's definitely value in running more complex tests, but I would say that if you're not running simpler, more frequent load and stress tests, then you should probably start there before you go for the large-scale, complex tests that closely simulate reality. The simple tests are the low-hanging fruit here:
- Running simple tests means each test requires less time and effort
- Less time and effort per test means you can run more tests
- More tests give you more opportunity to iterate and optimize your test setup
- More and smaller tests also allow you to run tests earlier in the development process, catching problems earlier and reducing both the cost of fixing problems and the risk of slipping delivery dates
Running multiple, simpler tests will teach you a lot about how to go about the more complex, large-scale testing as well:
- You'll better understand what kind of traffic puts stress on your back end and what doesn't
- You'll see how your back end reacts and know where to look for, what logging functionality to use, and generally how to get the most out of your testing
This means that when the time comes to run a complex, large-scale test, you'll be better prepared to do that. The large, high-risk load test will become smaller and lower-risk than ever before.
Don't let perfect be the enemy of good
Realism is a good goal, but don't lose sight of the trade-offs. If you wait to test until you have a perfect simulation, you'll miss opportunities to fix problems. Look for ways to start with simple testing instead, and then build up to more complex simulations.
Keep in mind that testing early is not a panacea, either. Problems can creep in late in development, and continuous testing can avoid last-minute fixes.
Finally, remember that you have limited time and a demanding schedule. If you can get 80 percent of the results with 20 percent of the work, you'll have more time to tackle other pressing issues.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
