Setting up Code Coverage
Request early access
This page describes features that are in private beta. Request early access to get them sooner.
Code coverage is used in software development to measure the percentage of code lines, branches, or paths that are executed during code exercises, such as automated tests. The primary goal of measuring code coverage is to identify areas of a program that are not adequately exercised, allowing developers to write additional tests to increase coverage and improve the overall reliability and quality of the software.
Why use Qlty for code coverage?
Integrating code coverage reporting provides a comprehensive view of your codebase’s health by displaying coverage data alongside your static analysis results. Once you’ve set up code coverage reporting, you can:
View code coverage metrics alongside other quality metrics like Maintainability or Duplication, all in one place.
Toggle between viewing code issues and code coverage line-by-line within the same source listings.
Enforce code coverage standards for PRs by configuring explicit Quality Gates.
We accept code coverage data from virtually any source, including locally run tests or your continuous integration (CI) service, and support a variety of programming languages and coverage formats, such as Ruby, JavaScript, Go, Python, PHP, Java, and more.
Setup
To upload a coverage report, you’ll need a previously generated code coverage report, a Qlty coverage token, and the Qlty CLI.
You must instrument the mechanics over which to exercise your code base and collect coverage metrics, such as with automated testing for example, and provide the final coverage report to Qlty, that’s the entry port to integrate with Qlty.
General Procedure
The general procedure for uploading code coverage reports is as follows:
Locate the coverage report generated by your code coverage tool. This report is typically part of the output from running automated tests, such as unit tests, configured to monitor code coverage.
Retrieve your Qlty coverage token
from your project’s settings page under the Code Coverage
section.
Use the Qlty CLI to upload your report:
- Upload your code coverage report
Checkout the coverage publish
command’s documentation for more information.
The Qlty CLI integrates seamlessly into your CI/CD pipeline. For templates and examples to help you get started with popular CI/CD providers, refer to the Continuous Integration (CI) section.
How to treat coverage tokens
It’s advisable to treat coverage tokens as sensitive information whenever possible. Some CI/CD suites, like GitHub Actions, offer mechanisms to store sensitive information, such as secrets, which can be used in your workflows. This is an excellent place to store your Qlty code coverage token.
We recommend treating this token as sensitive information. However, if the token is leaked, the potential damage is minimal, as the primary risk is someone uploading dummy coverage reports to your projects.
Rotate a coverage token
In your project’s settings under the Code Coverage
page, you will find an option to refresh your coverage token. This will provide you with a fresh token and invalidate the previous one.
Supported Languages and Formats
Report Format Specification
When uploading a coverage report, the format is typically inferred from the file extension or contents. However, you can explicitly specify the format using the --report-format
option, for acceptable values query the Qlty CLI with qlty publish --help
.
Path fixing
For Qlty to accurately match a coverage report file or any referenced file in your Git repository, the paths must align. Specifically, the path specified in your coverage report for a file must be relative to your Git project’s root. However, this alignment is not always present, as some tools generate coverage reports with absolute paths or may include additional content in the paths.
To resolve this issue, both qlty coverage publish
and qlty coverage transform
commands support the following flags:
--add-prefix
: The prefix to add to file paths in coverage payloads.--strip-prefix
: The prefix to remove from absolute paths in coverage payloads. This makes paths relative to the project root, typically the directory where tests were run. Defaults to the current working directory.
Advanced configurations
Parallel testing
The Qlty CLI can merge parallelized coverage reports into a single cohesive report for submission to Qlty.
Configure your CI to store partial results from each parallel run. It’s recommended to sync and fetch files from S3 or use a temporary folder within your CI build. Then, utilize the qlty coverage publish
to merge and upload the combined coverage report.
- Execute your test suite and store partial results in a shared and accessible location.
- Use
qlty coverage publish PATH_TO_REPORT_1 PATH_TO_REPORT_2 PATH_TO_REPORT_3
to merge and upload a single cohesive coverage report to Qlty.
If you need to individually transform coverage reports, you can use qlty coverage transform
.
- Run
qlty coverage transform
on reports from each parallel instance and store the results in a shared accessible location. - Collect each transformed report and upload them using
qlty coverage publish coverage_report_1.jsonl coverage_report_2.jsonl
.
A successful qlty coverage transform PATH
run produces a single file named coverage.jsonl
by
default, following the JSON Lines text format.
Multiple Test Suites
The procedure described above for merging multiple coverage reports from parallel testing can also be applied to combine coverage reports from multiple test suites. For example, you can merge coverage metrics from your unit test suite with those generated by your integration test suite.
We have a private beta for a feature which supports tracking code coverage from different test suites as independent metrics. If you are interested in this, please contact us.