[testing] What is code coverage and how do YOU measure it?

What is code coverage and how do YOU measure it?

I was asked this question regarding our automating testing code coverage. It seems to be that, outside of automated tools, it is more art than science. Are there any real-world examples of how to use code coverage?

Code coverage has been explained well in the previous answers. So this is more of an answer to the second part of the question.

We've used three tools to determine code coverage.

1. JTest - a proprietary tool built over JUnit. (It generates unit tests as well.)
2. Cobertura - an open source code coverage tool that can easily be coupled with JUnit tests to generate reports.
3. Emma - another - this one we've used for a slightly different purpose than unit testing. It has been used to generate coverage reports when the web application is accessed by end-users. This coupled with web testing tools (example: Canoo) can give you very useful coverage reports which tell you how much code is covered during typical end user usage.

We use these tools to

• Review that developers have written good unit tests
• Ensure that all code is traversed during black-box testing

Just remember, having "100% code-coverage" doesn't mean everything is tested completely - while it means every line of code is tested, it doesn't mean they are tested under every (common) situation..

I would use code-coverage to highlight bits of code that I should probably write tests for. For example, if whatever code-coverage tool shows myImportantFunction() isn't executed while running my current unit-tests, they should probably be improved.

Basically, 100% code-coverage doesn't mean your code is perfect. Use it as a guide to write more comprehensive (unit-)tests.

Code coverage is simply a measure of the code that is tested. There are a variety of coverage criteria that can be measured, but typically it is the various paths, conditions, functions, and statements within a program that makeup the total coverage. The code coverage metric is the just a percentage of tests that execute each of these coverage criteria.

As far as how I go about tracking unit test coverage on my projects, I use static code analysis tools to keep track.

In the previous answers Code coverage has been explained well . I am just adding some knowledge related to tools if your are working on iOS and OSX platforms, Xcode provides the facility to test and monitor code coverage.

Reference Links:

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/07-code_coverage.html

https://medium.com/zendesk-engineering/code-coverage-and-xcode-6b2fb8756a51

Both are helpful links for learning and exploring code coverage with Xcode.

For Perl there's the excellent Devel::Cover module which I regularly use on my modules.

If the build and installation is managed by Module::Build you can simply run ./Build testcover to get a nice HTML site that tells you the coverage per sub, line and condition, with nice colors making it easy to see which code path has not been covered.

Code coverage is simply a measure of the code that is tested. There are a variety of coverage criteria that can be measured, but typically it is the various paths, conditions, functions, and statements within a program that makeup the total coverage. The code coverage metric is the just a percentage of tests that execute each of these coverage criteria.

As far as how I go about tracking unit test coverage on my projects, I use static code analysis tools to keep track.

For Perl there's the excellent Devel::Cover module which I regularly use on my modules.

If the build and installation is managed by Module::Build you can simply run ./Build testcover to get a nice HTML site that tells you the coverage per sub, line and condition, with nice colors making it easy to see which code path has not been covered.

Code coverage basically tells you how much of your code is covered under tests. For example, if you have 90% code coverage, it means 10% of the code is not covered under tests.

I know you might be thinking that if 90% of the code is covered, it's good enough, but you have to look from a different angle. What is stopping you from getting 100% code coverage?

A good example will be this:

if(customer.IsOldCustomer()) 
{
}
else 
{
}

Now, in the code above, there are two paths/branches. If you are always hitting the "YES" branch, you are not covering the "else" part and it will be shown in the Code Coverage results. This is good because now you know that what is not covered and you can write a test to cover the "else" part. If there was no code coverage, you are just sitting on a time bomb, waiting to explode.

NCover is a good tool to measure code coverage.

Code coverage has been explained well in the previous answers. So this is more of an answer to the second part of the question.

We've used three tools to determine code coverage.

1. JTest - a proprietary tool built over JUnit. (It generates unit tests as well.)
2. Cobertura - an open source code coverage tool that can easily be coupled with JUnit tests to generate reports.
3. Emma - another - this one we've used for a slightly different purpose than unit testing. It has been used to generate coverage reports when the web application is accessed by end-users. This coupled with web testing tools (example: Canoo) can give you very useful coverage reports which tell you how much code is covered during typical end user usage.

We use these tools to

• Review that developers have written good unit tests
• Ensure that all code is traversed during black-box testing

Code coverage is simply a measure of the code that is tested. There are a variety of coverage criteria that can be measured, but typically it is the various paths, conditions, functions, and statements within a program that makeup the total coverage. The code coverage metric is the just a percentage of tests that execute each of these coverage criteria.

As far as how I go about tracking unit test coverage on my projects, I use static code analysis tools to keep track.

For Perl there's the excellent Devel::Cover module which I regularly use on my modules.

If the build and installation is managed by Module::Build you can simply run ./Build testcover to get a nice HTML site that tells you the coverage per sub, line and condition, with nice colors making it easy to see which code path has not been covered.

For PHP you should take a look at the Github from Sebastian Bergmann

Provides collection, processing, and rendering functionality for PHP code coverage information.

https://github.com/sebastianbergmann/php-code-coverage

Just remember, having "100% code-coverage" doesn't mean everything is tested completely - while it means every line of code is tested, it doesn't mean they are tested under every (common) situation..

I would use code-coverage to highlight bits of code that I should probably write tests for. For example, if whatever code-coverage tool shows myImportantFunction() isn't executed while running my current unit-tests, they should probably be improved.

Basically, 100% code-coverage doesn't mean your code is perfect. Use it as a guide to write more comprehensive (unit-)tests.

Complementing a few points to many of the previous answers:

Code coverage means, how well your test set is covering your source code. i.e. to what extent is the source code covered by the set of test cases.

As mentioned in above answers, there are various coverage criteria, like paths, conditions, functions, statements, etc. But additional criteria to be covered are

1. Condition coverage: All boolean expressions to be evaluated for true and false.
2. Decision coverage: Not just boolean expressions to be evaluated for true and false once, but to cover all subsequent if-elseif-else body.
3. Loop Coverage: means, has every possible loop been executed one time, more than once and zero time. Also, if we have assumption on max limit, then, if feasible, test maximum limit times and, one more than maximum limit times.
4. Entry and Exit Coverage: Test for all possible call and its return value.
5. Parameter Value Coverage (PVC). To check if all possible values for a parameter are tested. For example, a string could be any of these commonly: a) null, b) empty, c) whitespace (space, tabs, new line), d) valid string, e) invalid string, f) single-byte string, g) double-byte string. Failure to test each possible parameter value may leave a bug. Testing only one of these could result in 100% code coverage as each line is covered, but as only one of seven options are tested, means, only 14.2% coverage of parameter value.
6. Inheritance Coverage: In case of object oriented source, when returning a derived object referred by base class, coverage to evaluate, if sibling object is returned, should be tested.

Note: Static code analysis will find if there are any unreachable code or hanging code, i.e. code not covered by any other function call. And also other static coverage. Even if static code analysis reports that 100% code is covered, it does not give reports about your testing set if all possible code coverage is tested.

In the previous answers Code coverage has been explained well . I am just adding some knowledge related to tools if your are working on iOS and OSX platforms, Xcode provides the facility to test and monitor code coverage.

Reference Links:

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/07-code_coverage.html

https://medium.com/zendesk-engineering/code-coverage-and-xcode-6b2fb8756a51

Both are helpful links for learning and exploring code coverage with Xcode.

Code coverage has been explained well in the previous answers. So this is more of an answer to the second part of the question.

We've used three tools to determine code coverage.

1. JTest - a proprietary tool built over JUnit. (It generates unit tests as well.)
2. Cobertura - an open source code coverage tool that can easily be coupled with JUnit tests to generate reports.
3. Emma - another - this one we've used for a slightly different purpose than unit testing. It has been used to generate coverage reports when the web application is accessed by end-users. This coupled with web testing tools (example: Canoo) can give you very useful coverage reports which tell you how much code is covered during typical end user usage.

We use these tools to

• Review that developers have written good unit tests
• Ensure that all code is traversed during black-box testing

Code coverage basically tells you how much of your code is covered under tests. For example, if you have 90% code coverage, it means 10% of the code is not covered under tests.

I know you might be thinking that if 90% of the code is covered, it's good enough, but you have to look from a different angle. What is stopping you from getting 100% code coverage?

A good example will be this:

if(customer.IsOldCustomer()) 
{
}
else 
{
}

Now, in the code above, there are two paths/branches. If you are always hitting the "YES" branch, you are not covering the "else" part and it will be shown in the Code Coverage results. This is good because now you know that what is not covered and you can write a test to cover the "else" part. If there was no code coverage, you are just sitting on a time bomb, waiting to explode.

NCover is a good tool to measure code coverage.

For Perl there's the excellent Devel::Cover module which I regularly use on my modules.

If the build and installation is managed by Module::Build you can simply run ./Build testcover to get a nice HTML site that tells you the coverage per sub, line and condition, with nice colors making it easy to see which code path has not been covered.

Complementing a few points to many of the previous answers:

Code coverage means, how well your test set is covering your source code. i.e. to what extent is the source code covered by the set of test cases.

As mentioned in above answers, there are various coverage criteria, like paths, conditions, functions, statements, etc. But additional criteria to be covered are

1. Condition coverage: All boolean expressions to be evaluated for true and false.
2. Decision coverage: Not just boolean expressions to be evaluated for true and false once, but to cover all subsequent if-elseif-else body.
3. Loop Coverage: means, has every possible loop been executed one time, more than once and zero time. Also, if we have assumption on max limit, then, if feasible, test maximum limit times and, one more than maximum limit times.
4. Entry and Exit Coverage: Test for all possible call and its return value.
5. Parameter Value Coverage (PVC). To check if all possible values for a parameter are tested. For example, a string could be any of these commonly: a) null, b) empty, c) whitespace (space, tabs, new line), d) valid string, e) invalid string, f) single-byte string, g) double-byte string. Failure to test each possible parameter value may leave a bug. Testing only one of these could result in 100% code coverage as each line is covered, but as only one of seven options are tested, means, only 14.2% coverage of parameter value.
6. Inheritance Coverage: In case of object oriented source, when returning a derived object referred by base class, coverage to evaluate, if sibling object is returned, should be tested.

Note: Static code analysis will find if there are any unreachable code or hanging code, i.e. code not covered by any other function call. And also other static coverage. Even if static code analysis reports that 100% code is covered, it does not give reports about your testing set if all possible code coverage is tested.

Code coverage is simply a measure of the code that is tested. There are a variety of coverage criteria that can be measured, but typically it is the various paths, conditions, functions, and statements within a program that makeup the total coverage. The code coverage metric is the just a percentage of tests that execute each of these coverage criteria.

As far as how I go about tracking unit test coverage on my projects, I use static code analysis tools to keep track.

Just remember, having "100% code-coverage" doesn't mean everything is tested completely - while it means every line of code is tested, it doesn't mean they are tested under every (common) situation..

I would use code-coverage to highlight bits of code that I should probably write tests for. For example, if whatever code-coverage tool shows myImportantFunction() isn't executed while running my current unit-tests, they should probably be improved.

Basically, 100% code-coverage doesn't mean your code is perfect. Use it as a guide to write more comprehensive (unit-)tests.

Code coverage has been explained well in the previous answers. So this is more of an answer to the second part of the question.

We've used three tools to determine code coverage.

1. JTest - a proprietary tool built over JUnit. (It generates unit tests as well.)
2. Cobertura - an open source code coverage tool that can easily be coupled with JUnit tests to generate reports.
3. Emma - another - this one we've used for a slightly different purpose than unit testing. It has been used to generate coverage reports when the web application is accessed by end-users. This coupled with web testing tools (example: Canoo) can give you very useful coverage reports which tell you how much code is covered during typical end user usage.

We use these tools to

• Review that developers have written good unit tests
• Ensure that all code is traversed during black-box testing

Just remember, having "100% code-coverage" doesn't mean everything is tested completely - while it means every line of code is tested, it doesn't mean they are tested under every (common) situation..

I would use code-coverage to highlight bits of code that I should probably write tests for. For example, if whatever code-coverage tool shows myImportantFunction() isn't executed while running my current unit-tests, they should probably be improved.

Basically, 100% code-coverage doesn't mean your code is perfect. Use it as a guide to write more comprehensive (unit-)tests.

Code coverage basically tells you how much of your code is covered under tests. For example, if you have 90% code coverage, it means 10% of the code is not covered under tests.

I know you might be thinking that if 90% of the code is covered, it's good enough, but you have to look from a different angle. What is stopping you from getting 100% code coverage?

A good example will be this:

if(customer.IsOldCustomer()) 
{
}
else 
{
}

Now, in the code above, there are two paths/branches. If you are always hitting the "YES" branch, you are not covering the "else" part and it will be shown in the Code Coverage results. This is good because now you know that what is not covered and you can write a test to cover the "else" part. If there was no code coverage, you are just sitting on a time bomb, waiting to explode.

NCover is a good tool to measure code coverage.

For PHP you should take a look at the Github from Sebastian Bergmann

Provides collection, processing, and rendering functionality for PHP code coverage information.

https://github.com/sebastianbergmann/php-code-coverage

Code coverage basically tells you how much of your code is covered under tests. For example, if you have 90% code coverage, it means 10% of the code is not covered under tests.

I know you might be thinking that if 90% of the code is covered, it's good enough, but you have to look from a different angle. What is stopping you from getting 100% code coverage?

A good example will be this:

if(customer.IsOldCustomer()) 
{
}
else 
{
}

Now, in the code above, there are two paths/branches. If you are always hitting the "YES" branch, you are not covering the "else" part and it will be shown in the Code Coverage results. This is good because now you know that what is not covered and you can write a test to cover the "else" part. If there was no code coverage, you are just sitting on a time bomb, waiting to explode.

NCover is a good tool to measure code coverage.