Code Coverage Metrics
Request early access
This page describes features that are in private beta. Request early access to get them sooner.
Qlty’s code coverage reporting is built around three metrics: Total Coverage, Total Coverage Delta, and Diff Coverage.
Total Coverage
Total Coverage is based on the idea of line coverage which indicates whether a line of source code was executed during the automated test suite. This is expressed as a percentage using the formula:
Lines of code that are not executable, like comments and blank lines, are excluded from the percentage calculation. The determination of which lines of code are considered executable is made by the code coverage tool you are using. Examples include Istanbul for JavaScript and Simplecov for Ruby.
Total Covergage acts a snapshot or a point-in-time metric. Because it considers the entire codebase, it is likely to change slowly in large codebases due to a large denominator.
Currently we do not support branch coverage, which looks at whether each side of a boolean expression been evaluated.
Total Coverage Delta
By comparing the Total Coverage as of two different points in time, we can compute a Total Coverage Delta. This is done through subtraction:
In the context of a pull request, we compute the Total Coverage Delta by comparing the Total Coverage of the head commit compared to the coverage of the PR’s merge base. (In order for this calculation to be possible, we must have previously received a code coverage report for the merge base.)
Diff Coverage
Diff Coverage is a way to look at the absolute coverage of new or modified code. Since it is bulit on the concept of code changes, it can only be calculated in the context of a diff (typically a pull request).
You can think of Diff Coverage as a filtered version of Total Coverage. While Total Coverage always considered the coverage of the entire project, the Diff Coverage metric focuses exclusively on the lines of code that have been modified. The formula is:
In essence, it is the coverage of the green or right side lines in the diff. Since Diff Coverage only considers those lines, it often varies widely between pull requests. For example, a pull request that adds a single line of untested code will have a Diff Coverage of 0%.
By maintaining a high Diff Coverage on each pull request, your Total Coverage is likely to increase over time.
Coverage ratings
Total Coverage evolves over time and serves as a signal of codebase health, although its significance can vary depending on the nature of the project. For projects with a medium or long lifespan, coverage might not always be a priority. However, a high Total Coverage number is generally a positive indicator.
Qlty offers some absolute numbers to provide guidance. Your project’s Total Coverage number will determine its Qlty Coverage Grade, which will be a key metric visible on your project’s dashboard.
Quality Gates
GitHub has a robust system for commit statuses that is used for external systems to report information about code changes in a simple go/no-go form.
You can configure Qlty to report one of the following commit statuses:
Qlty Coverage: Displays the total code coverage from the automated test suite. This status can be configured to return a failure if the coverage decreases compared to the base branch.
Qlty Diff Coverage: Displays the code covergage of the new or changed lines in the pull request. This status can be configured to return a failure if the coverage of the modified code is below a threshold.