Code Coverage vs Test Coverage; Which Is Better? | LambdaTest

Image for post
Image for post

Test Coverage and Code coverage are the most popular methodologies for measuring the effectiveness of the code. Though these terms are sometimes used interchangeably since their underlying principles are the same. But they are not as similar as you may think. Many times, I have noticed the testing team and development team being confused over the use of these two terminologies. Which is why I thought of coming up with an article to talk about the differences between code coverage and test coverage in detail.

Code Coverage vs Test Coverage: How Do They Differ?

Test coverage : includes testing the features implemented as a part of the Functional requirements specification, software requirements specification, and other required documents. For example, if you are to perform cross browser testing of your web application to ensure whether your application is rendering well from different browsers or not? Your test coverage would be around the number of browsers + OS combinations over which you have validated browser compatibility of your web application.

With the understanding of the basic difference between code coverage vs test coverage, let us jump into further details around code coverage & test coverage.

Understanding Code Coverage In-Depth

Why You Should Perform Code Coverage?

As the product development progresses, new features, as well as fixes (to the bugs raised during testing) are added to the release cycle. This means that the test code may also require changes in order to keep it updated with the software changes made during development. It is important that the testing standards which were set during the start of the project are maintained with subsequent release cycles. Code coverage can be used to make sure that your tests are meeting those standards and the best quality code goes into the production phase.

Higher the percentage of code coverage; lower are the chances of having undetected bugs. In some organizations, the quality team sets the minimum amount of code coverage that needs to be achieved before the software is pushed to the production phase. The primary reason for the same is to reduce the probability of bugs being detected at a later stage of product development.

How To Perform Code Coverage?

  • Branch coverage — Branch coverage also called decision coverage is used to ensure that every possible branch used in a decision-making process is executed. For example, if you are incorporating a fall back for cross browser compatibility using an If…An else conditional statement or by a Do…While statement in your code, as a part of coverage; you have to ensure that all the branches i.e. If, Else, Do, and While are tested by providing appropriate inputs to make a cross browser compatible website.
  • Function coverage — Function coverage ensures that necessary functions (especially exported functions/APIs) are tested. This should also include testing the functions with different types of input parameters as that would also test the logic used in the functions. Once all the functions in the code are tested, function coverage would be 100%.
  • Statement Coverage — This is an important code coverage methodology in which test code has to be written in a manner that every executable statement in the source code is executed at least once. This also includes corner cases or boundary cases.
  • Loop Coverage — This approach is to ensure that every loop in the source is executed at least once. There might be some loops that may execute based on results that you achieved at runtime, it is important to test such loops as well in order to make the code foolproof.

For checking the code coverage, an approach called instrumentation is used. Instrumentation can be used to monitor the performance, insert trace information, and diagnose any kind of errors in the source code. There are different kinds of instrumentation and there could be a minimal performance (and timing) overhead depending on the instrumentation approach being used.

Types of Instrumentation

  • Code instrumentation — Here the source code is compiled after the addition of instrumentation statements. The compilation should be done using the normal toolchain, successful compilation results in the generation of instrumented assembly. For example, in order to check the time taken to execute a particular function in your code, you can add instrumentation statements in Start & End of the function.
  • Runtime instrumentation — Contrary to the code instrumentation approach, here the information is collected from the runtime environment i.e. when the code is under execution.
  • Intermediate code instrumentation — In this type of instrumentation, an instrumented class is generated by the addition of byte codes to the compiled class files.

Depending on your test requirement, you should choose the right code coverage tool and the best instrumentation approach supported by the tool.

Tools for Code Coverage

  1. Coverage.py — It is a code coverage tool for Python. As the name suggests, it analyzes your source code and identifies the percentage of code that was executed. It is developed in Python. It is free of cost and you can refer https://coverage.readthedocs.io/en/coverage-4.4.1/index.html for more information.
  2. Serenity BDD — Supporting Java & Groovy programming languages, Serenity BDD is a popular open-source library that is primarily used for writing excellent quality acceptance tests faster. You can make use of stories & epics for the tests and the code coverage is computed for these stories & epics. Due to this, the test reports that are generated are more illustrative and narrative in nature. You can map those automated tests back in your requirements. It can be used with JUnit, Cucumber, and JBehave. Serenity BDD can be easily integrated with Maven, Cradle, JIRA, and Ant. If you are using the Selenium WebDriver or Selenium Grid framework for automation testing, choosing Serenity BDD can be a huge advantage due to its compatibility with Selenium. You can refer https://www.thucydides.info/#/documentation for further information.
  3. JaCoCo — JaCoco is a code coverage tool for Java. Though there are other options like Cobertura & EMMA, these tools were deprecated since there was no update for a long time. The JaCoCo tool is a part of the Eclipse Foundation and it replaced the EMMA code coverage tool in Eclipse. Apart from the active development of JaCoCo, another advantage of using it is seamless integration with CI/CD & project management tools like Maven, Jenkins, Gradle, etc. Please visit official site for more information.
  4. JCov — JCov is a test framework agnostic code coverage tool. It can be effortlessly integrated with Oracle’s test infrastructure — JavaTest & JTReg. Though it is not in active development, support for on-the-fly instrumentation & offline instrumentation are the major advantages of using JCov. Please visit https://wiki.openjdk.java.net/display/CodeTools/jcov for further information.
  5. PITest — Most of the code coverage tools check the code for branch coverage, statement coverage, loop coverage, etc. and give out the coverage results. Though that information is useful to improve the quality of your test code, the test does not tell you how good the automation tests were in discovering bugs. This is where mutation testing can be very helpful. PITest is a very popular code coverage tool that is used for mutation testing for Java & JVM. It does the job of mutation testing by modifying your test code and the unit tests are now performed on this modified code. If bugs are discovered using this code i.e. after extra code is added by PITest, the unit test is foolproof; else it needs changes as the issues were left undiscovered. PITest is easy to use, fast and is under active development. It also integrates with popular CI/CD tools which makes it much more useful. Please refer http://pitest.org/ for more information.

Understanding Test Coverage In-Depth

How To Perform Test Coverage?

Some of the test coverage mechanisms are below:

  • Unit Testing — This type of testing is performed at a unit level/module level. Bugs encountered at a unit level can be different from the issues encountered at the integration stage.
  • Functional Testing — In functional testing, the functions/features are tested against the requirements mentioned in the Functional Requirement Specification (FRS).
  • Integration Testing — It is also called as system testing since the software is tested on a system level. This type of testing is performed once all the required modules are integrated.
  • Acceptance Testing — It all depends on the result of acceptance testing whether the product would be released to the end consumer/customer. This is where the developers get a sign-off as a green pass from the testers and SMEs of the web-application, right before pushing the code changes from the Staging environment to the Production.

The other important point to note is that the purpose & meaning of test coverage can differ depending on the level at which testing is performed. It also depends on the type of product on which black-box testing is performed. The test coverage metrics for testing mobile phone would differ from metrics for e-commerce website testing. Some classifications are below:

  • Features coverage — Here the test cases are developed in a manner that there is maximum coverage of product features. For example, if a tester is assigned the job of testing a phone dialer application; he should make sure that the number being dialled is proper in length. If the testing is done in India, the mobile number should be maximum 10 digits; else it should flash an error. Similar to this, all the mandatory & optional features have to be tested according to a priority set by the product team.
  • Risks coverage — Every product/project requirement document has a section that mentions about the Risks & Mitigation associated with the project. Though some risks (e.g. changes in business dynamics) are beyond the scope of the planning/development/test team, there are some risks that need to be addressed during the testing phase. For example, while developing a business website, the server infrastructure should be set up in a manner that page access is very fast. Depending on the location (i.e. country, city, etc.) from where the website is accessed, the closest server should be chosen for loading the website; else the entire experience would be hampered. The test team should also perform a load test where performance test is done when multiple numbers of users are trying to access the website at the same time i.e. scenario where there is huge traffic on the website. If the results of these tests are not good, it can result in below-average user experience (which can be a huge risk).
  • Requirements coverage — Here tests are defined in a manner that there is maximum coverage of the product requirements mentioned in various Requirement Specification documents. For example, if you are testing a pre-installed SMS application, you need to make sure that the default language is set properly e.g. If the mobile is marketed in a country where English is not the primary language e.g. China; the default SMS language should be Chinese and for other customers e.g. India; the default language should be English.

Tools for Test Coverage

Though there are few instances where you have to write test code for achieving the test coverage requirement, you may still need to use some of the popular test frameworks in some cases. Two of the most popular test frameworks are:

JUnit — JUnit is the unit testing framework for Java. It can also be used for UI testing. It is open-source and considered important in the development of TDD (Test Driven Development). Developers & testers make use of JUnit to write & execute tests that are repetitive. This also makes it a popular framework for regressive testing. Please refer https://junit.org/junit5/ for more information.

Run You First Selenium Automation Testing Script with JUnit

Run You First Selenium Automation Testing Script with PyUnit

Code Coverage vs Test Coverage; Which One Should You Follow?

The million-dollar question is “Which one of these is best suited for your project”?. There is no definite answer to this question as the solution depends on the type & complexity of the project. In most cases, test coverage and code coverage are used since they are equally important in software projects.

Advantages of Test Coverage

  • As the tests performed as a part of the coverage are black-box in nature, executing those tests might not require much expertise.

Shortcomings Of Depending On Test Coverage

  • No concrete way of measuring test coverage. Hence, the coverage results largely depend on the domain competence of the tester who is performing the tests and may vary from one tester to another.

Advantages of Code Coverage

  • Irrespective of the type of tool being used (open-source, premium), setting up a code coverage tool should not take much time.
  • Helps in improving the code quality by capturing bugs in the code.

Shortcomings of Code Coverage

  • The methodology used by tools could be different hence; you may not be able to compare code coverage results of one tool to another.
  • Searching for the best-suited tool could be a big task as you need to compare & try features from those tools before selecting the best one that suits your project requirements.
  • There are very few tools that provide support for different programming languages e.g. Java, Python, C, etc. Hence, you may need to have more than one tool in case your team is using multiple programming languages (for test code development).

How Much Coverage Is Enough?

The test team should spend a good amount of time in understanding the overall requirements and prioritize the test activities. In order to keep track of the progress, they should have a checklist that should be updated on a periodic basis (at least after every release). It is important that the test team also has frequent communication with the Quality Assurance (QA) team since they have details about the target (test/code) coverage that has to be achieved for the product/project to be released to the client/customer.

To summarize, there is no specific thumb rule which mentions about the minimum percentage of code coverage or test coverage that needs to be achieved while testing the product.

How Do I Ensure Maximum Code Coverage And Test Coverage?

Coverage is a useful metric but its scope is limited. There is no ideal way in which you can measure the effectiveness of the efforts spent on testing. It is a known fact that 100% coverage is only a myth.

A holistic approach that encompasses different types of testing e.g. automation testing, integration testing, manual testing, cross browser testing, etc. would be extremely useful. It helps you measure the effectiveness of different tests and test systems in a single place.

Don’t Aim For A 100% Code Coverage

Conclusion

Originally published at LambdaTest

Image for post
Image for post

Written by

Product Growth at @lambdatesting (www.lambdatest.com)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store