3 test design principles to get you to continuous integration

Whether you're doing automated or manual testing, it's critical that any software checks that you make to validate your assumptions about code follow three key principles to ensure they are fully compatible with a continuous integration and delivery system.

1. Reliability

Any test case that you're going to run with any frequency must be reliable; that is, the test case cannot be flaky. Consider an automated check: In a continuous integration environment, this test case could run dozens or hundreds of times a day for a single team.

If a test is only 99% reliable (one false report in every 100 test executions), and you run it 200 times a day, then your team will be investigating false-positive failures at least twice daily. Multiply that by a unit test suite that can have tens of thousands of test cases, and the math becomes clear.

But what does reliability look like? A test case must take every precaution to avoid a false negative or false positive. It must be repeatable without outside human intervention; it must clean up after itself.

In a fully automated system, there generally isn't time for a human to, for example, drop tables on the SQL database every few test runs. Even a manually run test case must clean up after itself, because it is an unmanageable mental burden on the test executor to have a continuously shifting starting state.

Why is reliability so important? When developers must routinely waste time investigating false positives or negatives, they quickly lose faith in the automation solution and are likely to ignore real failures alongside the false ones.

2. Importance

In a continuous integration system, the most precious resource you have to spend is engineers' time. Engineers have grown to expect results quickly, and they are unwilling to wait for something they perceive as wasting time. So ensure that you're getting relevant results back as quickly as possible.

For example, there's no point in attempting to run unit tests on code that doesn't compile. And there's no point in running an API-level integration test suite if the unit tests on an underlying package don't pass. You're assured that the code under test will have to change, so why waste time on a test run that is guaranteed to be thrown away?

Figure 1. In this modified testing pyramid, unit tests make up the foundation of your testing strategy, integration tests validate across boundaries, and specialty tests at the top capture any slow or complex testing. Source: Melissa Benua. 

Always run the most important test cases as quickly as possible, and always run your fastest tests first. Those are nearly always your unit tests; a typical unit test executes in microseconds and can generally be run in parallel. In my continuous integration systems, I can usually process through tens of thousands of unit tests in around 90 seconds.

An integration test is a test that crosses boundaries, usually including at least an HTTP or other machine-to-machine boundary. By definition, these test cases execute in milliseconds and are several orders of magnitude slower than are unit tests.

Finally, a specialty test is anything that's significantly slower than an integration test (such as an end-to-end automated UI test) or one that requires human intervention or interpretation that slows down the overall reporting of results.

While tests slower than a unit test certainly have value, and absolutely have a place in a continuous integration system, that place is after the faster and more reliable tests have run.

3. Specificity

A good test case should individually do as little as possible to produce a pass/fail outcome as quickly as possible. If you had infinite time in which to execute a test run, overlapping coverage and redundant tests wouldn't be very important.

But if you only have a budget of five minutes for an entire integration test pass, for example, and every integration test case takes 10 milliseconds, then you have time to run only 30,000 test cases. And if you are doing a UI-based test that is closer to 1 second per test, then you have time for only 300 test cases.

Once you acknowledge the reality of an upper bound to the number of test cases you may run, you can consider exactly how to spend those resources.

test_explorer.png

Figure 2. Always name test cases in a clear, descriptive manner. As you can see above, that makes it easier to diagnose failures. Source: Melissa Benua.

Every test case should be a clear answer to a clear question, the combination of which adds up to a test suite that will give a clear answer about the full set of functionalities under test. Clearly named, atomic test cases should make it easy to locate the potential cause of a test failure, and also make it easy to understand at a glance what has and what has not been tested.

When test cases are clear and atomic, it becomes easy to find coverage overlaps, and thus candidates for deletion.

Most likely all of A, B, and C will have a combination of 100x of unit tests and 10x of integration tests, depending on their specific system architecture boundaries. While C may require a logged in user, the purpose of the suite is not to ensure that you can create a user or update the user's settings.

Creation and setting changes are incidental functionalities that are likely called during test suite setup; if they fail then, do not attempt to test any additional functionalities. Given this known order of precedence, you also want to ensure that test suites are run in A, B, C order, since you know that C is dependent on functionality in A and B. It's useless to try to execute C if either A or B is known to not work.

Now apply the principles to improve stability

 If as you've moved to continuous delivery your automated or manual tests are causing your software to be less stable, the steps above are for you. Follow these three key principles and your tests will always be compatible with your organization's continuous delivery efforts.

Melissa Benua will be presenting at the the STAREAST virtual online conference, which takes place from May 4-7, 2020. Her tutorials include Continuous Testing Using Containers and Advanced Test Design for CI/CD.  She's also presenting on Fuzz Testing. Click here for details.