Application tests are useful because they help you go faster. They help you to make changes to your code without fear of breaking existing functionality. That said, I’ve seen teams get slowed down by their approach to testing. Here's how you can avoid that fate.
Remember that tests aren't free
All code incurs a maintenance cost over time. This is as true for test code as it is for your production code. There are two maintenance costs associated with tests.
- The cost of working with the code. Every time developers want to add new functionality, they must first navigate the existing test suite to find an appropriate location and pattern for the new tests. When refactoring code or fixing bugs, the developer needs to understand the existing tests and ensure that those tests still safeguard the desired behavior after the changes.
- The running cost of your tests. To get value from tests, it’s crucial to run them often, ideally on every commit. If they’re slow, or they break unexpectedly, they will become time-consuming. I’ve seen some teams give up on testing altogether because of slow and brittle tests.
Both costs can be reduced. Consider these two steps:
- Trim down and reorganize your test code.
- Speed up test suites or run them in parallel.
But before you move forward, you must weigh the cost of doing this against the value that each test brings. First figure out which tests are earning their keep and which ones aren't. Then focus only on the ones that are, and spend the time you've gained on shipping new features.
Don't get tangled in unit tests
Unit tests often make refactoring more difficult, because they can be specific to a particular implementation or heavily reliant on mocks of the rest of the system. David Heinemeier Hansson wrote about the difficulty of working with such tests in his post, TDD is Dead, a few of years ago. I’ve personally seen developers spend hours fixing broken unit tests after a refactoring, only to realize that the functionality doesn’t actually work as intended anymore.
When a unit test makes refactoring harder, delete it and write an acceptance-level test instead.
These are tests which exercise a full path through your system, often at the API level, and can ensure that the actual functionality you care about works.
Make sure your tests fail
On many occasions, I've started working on an existing codebase, only to find dozens of tests that would never fail, even if the code under test were deleted completely. This is usually due to too much mocking, a poor understanding of the framework used by the code under test, or a poor understanding of the testing library being used.
A common mistake in JavaScript codebases is misunderstanding asynchronous code and ending up with test assertions that will never run in the case of failure.
Make sure that your test will actually fail when functionality breaks. Writing your tests before writing the implementation is often a useful way to ensure this outcome.
Stop patching your UI tests
Test suites that exercise an application’s user interface (UI) are notorious for being brittle and providing false negatives. This happens because the tests are required to manage and work around the real, often unexpected, behavior of a UI, which runs as a separate process.
These kinds of tests can be extremely valuable. They’re able to provide a high level of confidence that functionality works as expected, because they exercise the full application.
The trick is not to maintain a huge suite of tests. Figure out which ones add value. For the rest, follow these two techniques.
- Push tests down a layer. In a web application, for example, you can test at the API level instead of using a browser. You can still write BDD-style tests in this way, describing the interaction with the data in the API, instead of the interaction with a UI. This lets you focus on testing the core behavior of your application without worrying about UI quirks.
- Monitor your application’s behavior. Create dashboards that help you see the application’s behavior in production. Pick key metrics that indicate whether something has gone wrong, and set up alerts so the team knows when they do. Replacing tests with monitoring won’t prevent regression defects, but it will allow you to address defects quickly, which may be enough for the majority of applications. Good monitoring also can help root out issues that UI tests would never catch, because UI tests are only focused on testing scenarios of which you’re already aware.
Don't test performance in a bubble
I've seen teams spend an enormous amount of time desperately running and re-running performance tests in their testing environments, to no avail. The reason? Load testing in test environments tends to be highly reliant on unreliable downstream systems, and test data that's difficult to set up. To make matters worse, the performance testing team is often stuck running these tests at night and over weekends so they don’t affect the work of other teams.
Rather benchmark new technologies or infrastructure in isolation. There are plenty of benchmarking modules that you can use. For example, it's easy to get started with Ruby benchmark module and Benchmark.js. Use such a library to put the code you’re concerned about through its paces.
You can compare the speed of an old third-party library with a new one, for example, and get a reliable, noise-free indication of how they perform against one another. You can perform similar tests at a much coarser level (e.g. hitting a simple status endpoint over HTTP) to compare different web servers, load balancers or even physical hardware.
Invest in production performance monitoring. Instead of running expensive load tests that don’t tell you anything, get some data about how your system is behaving in production. If you’re currently supporting X users and have an average response time of Y, use this data to figure out what the impact will be when your user base increases to 10X. Make sure you have good alerting in place so that you know when things go wrong (e.g. when your database server starts running out of memory). Invest in the ability to scale quickly.
Don't be afraid to delete tests
Tests are important, but they are not sacred. Go ahead and delete the ones that aren't helping. Every line of code adds to the complexity of a codebase. If a test adds complexity without adding a valuable safeguard for your functionality, then delete it. And don’t feel bad about it.
If you feel like a safeguard is missing after deleting a knot of unit tests, find a way to regain confidence in a simpler manner. Write an acceptance test, add some monitoring. Then ship something new.
Have you had tests slow you down? Do you have tips you can share? Post your comments and tips below.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
