A few years back, I was consulting on a project where I encountered an automated test suite that was in bad shape. It was a monstrous, constantly failing, long-running suite of tests that the team had created using Selenium. The codebase, which had been around for over 10 years, contained more than 1,000 tests, and the full regression suite took 18 hours to run. Worst of all, the tests failed 100% of the time.
The situation was grim, but after a lot of hard work and persistence, we turned the test suite around by focusing on three principles: Always create valuable, reliable, and fast tests.
The point of automated tests is to provide confidence that your application works as expected. When you have the right level of automation, you spend less time doing manual verification and more time writing code and shipping new features.
It’s easy to write new tests, which is good. But over time, without proper implementation and maintenance, your test suites will grow out of control. So how do you tame your tests and get them to a place where they can be as efficient as possible?
Every test should be valuable
The purpose of tests is to provide feedback that a specific feature is working as expected. This includes "happy path" and edge cases that ensure confidence in the area being tested.
One big problem on this team was a culture that rewarded quantity of tests over quality. The thinking was, the more tests you wrote, the better a tester you must be. It was easy to write tests, so engineers wrote lots of them to hit this perceived goal mandated by management. And over years and years, the number of tests grew out of control.
Before it could solve the problem, the team had to agree to only retain tests that were truly valuable. Going forward, it would never just automate something because it could be; it had to add value. Using this rule, it could decide which tests it should automate and which it shouldn't. This simple rule will have a huge impact on how your test suite looks over time.
Here are a few other tips for creating valuable tests:
- Re-evaluate and analyze so that each test's purpose and level of focus is clear.
- Collaborate with your team on testing so that everyone is aligned on the value that each test delivers.
- Emphasize quality versus quantity: Write tests that test the right thing; don't just test everything.
All tests must be reliable
All tests should have a consistent result every time you run them. This sounds straightforward, but sometimes unexplainable things happen to tests. Under the pressure of delivering new software, test engineers sometimes push test upkeep aside, and flaky or inconsistent results happen far too often.
If your team doesn’t trust the output of its tests, then those tests are useless. Throw them away. It’s critical that tests have consistent passing results. If they don't, reconsider your testing strategy.
Tips for creating reliable tests include:
- Separate tests that have the same result every time from those that don’t. Fix the ones that don't and then put them back one by one as you get them to pass.
- Rerun failures automatically as a failsafe, and to avoid having to run the entire test suite over again.
- Use specific test selectors to help avoid CSS changes that break tests and indicate that the element is used specifically for testing.
Your tests should be as fast as possible
As your test suite grows, the tests will take more time to finish, and build times will lengthen. Tests that are really slow cause a long feedback loop. That's bad, because it means you're not shipping code as fast. And no one wants to sit around waiting for tests to finish. So whenever possible, look for opportunities to make your tests run faster.
Tips for creating fast tests include:
- Parallelize tests so you can run multiple test classes at the same time.
- Avoid testing in the UI, and look for opportunities to automate features at a lower level (API, integration, unit).
- Run your tests headless to avoid having to wait for browser pages to load and render.
Next step: Start optimizing your tests
By following these three principles, we were able to whip the test suite into shape. While the process was a struggle, I always look back at the lessons I learned from the project appreciatively. By adding these principles to my toolkit I now know how to approach seemingly unmanageable test suites.
Here are a few other things I've learned that you should consider as you start the process of streamlining your test suites:
- There are always tradeoffs. You can't always make all the changes you want at one time, so prioritize which of the three principles is most important to you and start there.
- Testing is a team effort. Work together to figure out the most optimal testing strategy.
- It’s okay to start over. All test suites can be repaired, but it’s up to you and your team to decide if you’d rather spend time cleaning up or starting fresh. Just remember best practices along the way.
Following these key principles has served me well. Do you have your own, battle-tested tips for creating and maintaining a lean test suite? If so, please share by posting your suggestions below.
For more examples of how I've made this work in practice, attend my upcoming talk at SeleniumConf Berlin, where I'll expand on evaluating a test's value, parallelizing tests, and producing consistent results. My presentation will be live-streamed, and available on YouTube following the conference as well.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
