How Code Coverage Helps Validate Your Unit Tests

What Is Code Coverage?

Unit testing is an important addition to your development process to ensure that your production code is high quality and bug free.  But how do you know your unit tests are adequately validating the production code?  How do you know that you’ve tested all of the positive and negative test cases?  How do you know that there isn’t an uncaught exception for an obscure error case?  While not a complete answer to the quality of your unit tests, code coverage tools can be a huge help by telling you how much of your production code the unit tests are actually exercising.  There are several types of code coverage analysis and this article we’ll go over a few of the most common.

Line Coverage

The simplest type of code coverage analysis is line coverage.  This type of code coverage tells you if a specific line in the production code was executed or not during the test.  Here’s an example of line coverage:

    int doSomething( string str ) {
1        int a, b;
2        a = std::stoi(str);  b = a + 5;
3        if( b == 10 )
4            return 0;
5        else
6            return 1;
    }
// 6 lines, 6 covered = 100% coverage 

Statement Coverage

A slight variant to this is known as statement coverage.  Statement coverage takes things a step further to tell you if each statement was executed.  This type of code coverage will identify multiple statements on a particular line and will validate that each statement is executed.  The example below is the same code as above but the code coverage tool recognizes that there are two statements on the second line and validates that they are both executed.  

      int doSomething( string str ) {
1         int a, b;
2, 3      a = std::stoi(str);  b = a + 5;
4         if( b == 10 )
5             return 0;
6         else
7             return 1;
      }
// 7 statements, 7 covered = 100% coverage 

Branch Coverage

Another common type of code coverage is branch coverage.  This type of coverage tells you if all of the decision paths in the production code have been taken.  These are things like if/else,  or switch/case statements.  Arguably, this can also be determined from line and/or statement coverage by visually verifying that each section is executed.  The benefit of branch coverage is that it automatically tells you if all the control blocks in your code have been executed. Also, you can easily have a high line/statement coverage with a relatively low branch coverage.  Consider the  following example:

    int doSomething( string str ) {
1       int a = 0;
2       int b = 0;
3       try
        {
4           a = std::stoi(str);  
5           b = a + 5;
6           std::cout << “b = “ << b << std::endl;
        }
7       catch( const std::bad_alloc& e )
        {
8           return -1;
        }
9       if( b == 10 )
10          return 0;
11        else
12          return 1;
    }
// 12 statements , 11 covered = 92% coverage 

This code is being analyzed by statement coverage and has 92% coverage.  This doesn’t sound to bad.  But look at the same code analyzed with branch coverage:

    int doSomething( string str ) {
1       int a = 0;
        int b = 0;
        try
        {
2           a = std::stoi(str);  
            b = a + 5;
            std::cout << “b = “ << b << std::endl;
        }
        catch( const std::bad_alloc& e )
        {
3           return -1;
        }

        if( b == 10 )
4           return 0;
        else
5           return 1;
    }
// 5 control sections, 4 covered = 80% coverage 

 With branch coverage only four out of the five control sections are executed resulting in 80% coverage which is not nearly as good and would be a much more significant indicator that the unit tests are not testing the code thoroughly.

Additional Types of Coverage

 There are many additional types of code coverage that can be implemented:

  • Function Coverage – Gives the percentage of the functions in the production code that are executed during the testing.
  • Loop Coverage – Have all the loops in the code been executed at least once
  • Entry/Exit Coverage – Have all entries/exits of a function been executed
  • Parameter Value Coverage – Verify code execution with common range of values for the input parameters
  • Conditional Coverage – Verify each boolean statement is evaluated to both true and false.

 Many of these additional types of code coverage are typically only used with safety critical software as the the additional overhead and cost of the tools can outweigh the benefits they provide.  But if you’re not using any type of code coverage tools in your current development process then even simple line coverage would be a huge benefit in verifying that your unit tests are at least exercising the production in all the necessary test cases.  

Conclusion

 Unit testing along with other essential software development best practices (peer code reviews, object oriented design, static code analysis) is essential in making sure you have high quality and bug free production code.  Using code coverage analysis tools can help you ensure your unit tests are actually doing a thorough job and give you confidence in that safety net catching any bugs in the production code.