Micro Focus is now part of OpenText. Learn more >

You are here

Home/App Dev & Testing/Testing

You are here

Home/App Dev & Testing/Testing
Testing, done right

How to make your code bulletproof with property testing

public://webform/writeforus/profile-pictures/received_10208308343079540_0.jpeg
Jess Ingrassellino Engineering Manager, InfluxData
 

When people talk about code coverage, they usually mean statement coverage. The goal for some is to "cover" all possible statements in the codebase with tests. Yet a simple division program can have 100% statement coverage and still generate problematic results when asked to divide by zero.

The problem is that test engineers write mostly example-based tests where only one input scenario is tested. Property testing is a useful addition to your test suite because it runs one statement hundreds of times with different inputs. Property testing frameworks use almost every conceivable input that could break your code, such as empty lists, negative numbers, and really long lists or strings.

If your code survives a property test, you can be pretty sure it's resilient enough for production. Here's how property-based testing works in practice.

What is property-based testing?

Property-based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test. 

Property-based testing was popularized by the Haskell library QuickCheck. Tests that are similar to property-based tests include fuzz testing, or the use of example tables in ATDD/BDD-style testing.

Why use property-based testing

Given that most developers already do at least some unit testing, and software development engineers in test (SDETs) are often focused on other types of test tooling, one might wonder why they should add property-based tests to their strategic mix.

These tests are useful when a range of inputs need to be tested on a given aspect of a software property, or if you are concerned about finding missed edge cases.

For example, a retail website might have an items property whose value must always be greater than 0 but less than 10. A single property-based test for the items property could be written to test a negative value, some values within range, and a few values greater than 10.

Another, more complicated example might be a piece of medical software with a heart rate property. Business requirements might say that if the heart rate is out of normal range, a warning message should appear.

Because unit tests check single aspects of the code, several would need to be written to test every value that is in range, out of range, valid, or invalid. Property-based tests, in contrast, allows all of the desired inputs that should have a specific output to be tested in one test, allowing for more efficient test writing and running, and speedier test runs.

How to do property-based testing

Let's walk through an example of property-based tests. The scenario is a basic retail website with a property called items. This property has a lower boundary of 0 and an upper boundary of 10, including the number 10. If the value entered is anything other than an integer between 0 and 10, then an error should show.

Unit tests for this item might include a test for each of the following:

  • Assert 0 is valid
  • Assert 10 is valid
  • Assert some value between 0 and 10 is valid
  • Assert -1 is invalid
  • Assert 11 is invalid
  • Assert "some_string" is invalid
  • Assert 1.2 (or some decimal) is invalid

There are seven tests in this list already, and these test only the most obvious cases and not most of the potential edge cases. However, by using property-based tests, you can write one test to check all values that should work, and another test for each value type that should cause errors, using the following steps:

  1. Define the behaviors you are looking to test.

  2. Define the range of data you want to use to check against each outcome.

  3. Use a library in your code language to generate data sets (either random or specified) to use in your tests.

  4. Write and run tests.

Step 1: Define the behaviors

So, the business case for testing a field that takes the number of items is that the field should accept 0 to 10 items, inclusive of both values. The field should not accept any negative numbers, any numbers above 10, or any decimals. The field should not accept any strings of any kind, so testing special strings such as SQL injections might be a good idea.

Step 2: Determine the range of data needed to test behaviors

As noted in the scenario, traditional unit tests would require a test to be written for every single input. In PBT, tests are written for a group of inputs that yield the expected output.

To determine the tests that need to be written, the types of data groups must be defined. An easy way to do this for any scenario is to first write a test with a single input that should result in a desired output, and then generalize that input to a range of inputs.

Test group 1: Numeric values that should not yield an error (integers 1-10, inclusive)

Test group 2: Numeric values that should yield an error (negative integers, integers above 10, decimals)

Test group 3: Strings that should yield an error (characters, SQL)

Step 3: Generate your test data 

Part of property-based testing is being able to reuse the same test on a set of data. An alternative to QuickCheck is Hypothesis, a Python framework based on QuickCheck. Another property-based testing library option is ScalaTest, which is written in Scala and offers table-driven property checks—a method for generating several inputs to test against a given output. Both ScalaTest and Hypothesis are different from QuickCheck because their data is not randomized, which means that it is easier to define test logic and to repeat tests.

In the code examples in Step 4, I'll be using the fscheck library. Like QuickCheck, fscheck uses a random input approach, designed to be coded in C# for the .NET framework.

Step 4: Write and run tests

Depending on the code library you use, you can generate your test data and make sure that you are covering the test cases that you wish, as well uncovering any edge cases you might not have considered. Most of the libraries offer the ability to define data ranges and randomize data for tests. Also, some of the frameworks can seed the tests, so if a random input generated by your test method fails, it can be saved and reused in future tests to uncover other failures.

Here's a simple property test example, written in C# to support the FSCcheck library. Consider this function signature, which takes three integer inputs and determines if the result is a triangle.

private bool IsaTriangle(int sideA, intSideB, intSideC)

Recall the rules for triangles: Any two sides must add up to more than any individual side and no side can be negative. A classic approach to testing this would be to come up with examples. A more complex version might code the program twice, looping through the entire possible input space and comparing (and debugging) the results.

With property-based testing, we can define simple rules—for example, if the values are X, X*2, and X*3, the item is a triangle, and for X, X*10, and X*11, it is not.

[Property]
Public bool x_greater_than_yplusz_is_true(PositiveInt x)
{
  return (true==IsaTriangle(x*2,x*3,x*3);
}

[Property]
Public bool x_greater_than_yplusz_is_true(PositiveInt x)
{
  return (true==IsaTriangle(x,x*10,x);
}

[Property]
Public bool x_greater_than_yplusz_is_true(NegativeInt x)
{
  return (false==IsaTriangle(x*2,x*3,x*3);
}

You'll notice that the rules change when the inputs flip from positive to negative, the examples still don't cover zero, and there are a few more potential test cases to write. Still, with just three examples, we get a large amount of coverage written in a way that explains the rules, instead of just giving sample data points.

Where to find more information

Property-based testing and the frameworks that support it are available in many mainstream languages. In addition to QuickCheck, FSCheck, ScalaTest, and Hypothesis, Java has junit-quickcheck and Ruby has rantly. There are other libraries too, and a Google search for the "language of your choice + property based testing" will reveal more information about available libraries in specific languages.

If you’re looking to research property-based tests further, check out these articles:

Or share your experiences with property-based testing in the comments below.

Keep learning

Read more articles about: App Dev & Testing, Testing
Twitter Twitter Facebook Watsapp Email share