Code coverage types and pitfalls

Introduction

I introduced you to source code coverage. I explained the coverage standards such as C0, C1, and C2 in it, but there are other coverage standards according to the purpose and application.

Coverage standards, especially code coverage, are something you should be aware of when considering system quality, but some negative consequences can occur if you stick to covering too much.

In this article, we will introduce coverage standards at once, and also introduce examples that can be called “coverage traps” that you should be careful about when thinking about code coverage.

Complete code coverage

Focusing on code coverage, we will explain the characteristics of coverage while clarifying the specific code. This time, we would like to introduce the following 7 types of coverage.

1. Statement coverage (C0)
2. Decision coverage (C1)
3. Condition coverage
4. Condition / decision coverage
5. MC / DC
6. Combined condition coverage (C2)
7. Path coverage

1. Statement coverage (C0)

It is also called instruction coverage. The idea is that executing all statements in a program is directly linked to improving coverage.

Code example 1

How to make statement coverage 100%

If you execute the test case with the variable x = 0, all the statements (CommandA, B, C) will be executed. According to this idea, one pattern of test cases is enough.

2. Decision coverage (C1)

Also known as judgment coverage or branch coverage. The idea is to execute all judgment statements in the program to cover all the judgment results.

Code example 2

How to get 100% decision coverage

By executing two types of test cases, “when the variable x is” 0 “and” when it is not 0 “, both TRUE / FALSE of the condition of the if statement can be covered. According to this idea, two patterns of test cases are sufficient.

3. Condition coverage

It is also called condition coverage. The idea is to execute all patterns by covering each conditional expression in the judgment statement for all judgment statements in the program.

Code example 3

How to get 100% decision coverage

In this case, there are two types of judgment expressions, “x == 0” and “y <= 10”.

In the concept of decision coverage, it is OK if the true judgment can be covered for each conditional expression.

in short,

“X = 0, y = 100” … “Pattern (a): TRUE” and “Pattern (d): FALSE”

“X = 1, y = 0” … “Pattern (b): FALSE” and “Pattern (c): TRUE”

If you run these two test cases, the condition coverage will be 100%.

4. Condition/decision coverage

It is also called condition judgment coverage, decision/condition coverage, and judgment condition coverage. The idea is to cover both decision coverage and condition coverage.

In other words, it is possible to cover the difficulty that even if one of the condition decisions is covered, it is not always possible to cover the other. This is an important idea to improve the coverage of both conditions and decisions.

Code example 4

How to get 100% condition / decision coverage. In this way of thinking, it is necessary to cover the results of the true judgment and the judgment formula for each conditional expression.

So, in addition to the above, you need to consider the following:

therefore,

“X = 0, y = 0” … “Pattern (a): TRUE” and “Pattern (c): TRUE” → TRUE (A)

“X = 1, y = 100” … “Pattern (b): FALSE” and “Pattern (d): FALSE” → FALSE (D)

If you run these two test cases, the decision / condition coverage will be 100%.

5. MC / DC

Abbreviation for Modified Condition / Decision Coverage. It is translated as comprehensive improvement condition judgment.

Even if the above-mentioned condition/decision coverage is used, the coverage will be the same as the decision coverage, so MC / DC is used when more detailed and detailed coverage is required.

In particular, it is set to use this concept in safety standards for aviation equipment such as DO-178B (RTCA).

According to the definition of DO-178B, the following conditions must be met for 100% MC / DC coverage.

1. The judgment statement executes all results
2. Conditional expression executes all patterns
3. Each condition in the conditional expression alone affects the result

1. and 2. are the same criteria as condition/decision coverage, but you can achieve more detailed coverage by adding 3. to these two.

Code example 5

How to make MC / DC 100%

As with decision/condition coverage, the fact that each conditional expression covers the results of authenticity judgments and judgment expressions remains the same.

In addition to this, it is necessary to select a pattern in which the conditional expression alone affects the judgment result.

therefore,

“X = 0, y = 100” … “Pattern (a): TRUE” and “Pattern (d): FALSE” → TRUE (B)

“X = 1, y = 0” … “Pattern (b): FALSE” and “Pattern (c): TRUE” → TRUE (C)

“X = 1, y = 100” … “Pattern (b): FALSE” and “Pattern (d): FALSE” → FALSE (D)

If you run the three test cases, MC / DC will be 100%.

The remaining

“Pattern (a): TRUE” and “Pattern (c): TRUE” → TRUE (A)

As the judgment result does not change even if the truth of the conditional expression is changed, the condition of 3. of MC / DC is not satisfied. Therefore, it can be determined that it is not necessary to include it in the test case.

6. Composite condition coverage (C2)

Also known as multi-condition coverage. The idea is to execute all combinations of conditional expressions.

This is the most comprehensive standard of coverage for a single judgment statement, but it also has the disadvantage that the number of test cases increases extremely as the number of conditional expressions increases.

Code example 6

How to make MC / DC 100%

Since we need to cover all combinations of conditional expressions, we need to execute all four patterns shown in the table above.

7. Path coverage

The coverage of 1 to 5 introduced so far expresses the coverage rate for a single judgment sentence. On the other hand, path coverage is the standard for covering all routes (paths) for a certain range of a program (functions, classes, etc.).

Therefore, as the number of judgment statements in the program increases, the number of test cases will increase dramatically. In particular, in the case of a program that includes loops such as for statements and while statements, it is practically difficult to cover 100%.

Code coverage pitfalls

We found that there are various types of coverage, and the quality and guarantees that can be achieved differ depending on the viewpoint from which the coverage is enhanced. Bringing coverage close to 100% is a very important aspect of improving test case quality, but it’s not enough.

This is because even if you achieve improved coverage, looking back at the test cases and looking at what you can guarantee, the quality required for the system may not be sufficient.

For example, suppose you have the following statement:

This example is the statement “substitute the result of b ÷ c into a”.

So, for example, if you run a test case that sets b to 10 and c to 5, the coverage will be 100%.

However, if c is 0, you will get a division by zero error. In other words, it can be said that the error cannot be taken into consideration just by the coverage becoming 100%, and sufficient quality assurance cannot be performed.

It is necessary to ensure that c is not 0 to cause an error somewhere, but if you are only focusing on coverage, you may not be able to confirm this point of view.

Also, consider the code example given so far in the coverage description.

Depending on the programming language and structure, if “x == 0” is true, the “y <= 10” part may not be determined. Such a mechanism is called “lazy evaluation”.

In this case, the coverage can be set to 100% only in the part of “x == 0” without evaluating the value of y, so it may be insufficient for the actual test. increase.

In order not to fall into such a “coverage trap”, it may be better not to focus too much on coverage. It is necessary to confirm whether the content of the test case created from the viewpoint of coverage is sufficient for the quality assurance of the entire system.

In conclusion

In this article, I’ve explained the typical types of coverage with specific code examples, and then explained the points to be aware of when dealing with coverage. Coverage, especially code coverage, is a very important point of view when conducting a white-box test that is comprehensive, and centered on conditional expressions and their judgment results.

How to choose test cases depends on how comprehensive you should be, so you will need to choose a coverage method based on those characteristics.

However, there are surprisingly many perspectives that cannot be covered by the concept of coverage. It can be said that it is necessary to check every time whether the test case that achieved coverage holds the viewpoint that should be guaranteed due to the nature of the system.

If you ever want to know about similar things, check out the Facebook page Maga Techs