Setting up Code Coverage

Request early access

This page describes features that are in private beta. Request early access to get them sooner.

Code coverage is used in software development to measure the percentage of code lines, branches, or paths that are executed during code exercises, such as automated tests. The primary goal of measuring code coverage is to identify areas of a program that are not adequately exercised, allowing developers to write additional tests to increase coverage and improve the overall reliability and quality of the software.

Why use Qlty for code coverage?

Integrating code coverage reporting provides a comprehensive view of your codebase’s health by displaying coverage data alongside your static analysis results. Once you’ve set up code coverage reporting, you can:

View code coverage metrics alongside other quality metrics like Maintainability or Duplication, all in one place.

Toggle between viewing code issues and code coverage line-by-line within the same source listings.

Enforce code coverage standards for PRs by configuring explicit Quality Gates.

We accept code coverage data from virtually any source, including locally run tests or your continuous integration (CI) service, and support a variety of programming languages and coverage formats, such as Ruby, JavaScript, Go, Python, PHP, Java, and more.

Setup

To upload a coverage report, you’ll need a previously generated code coverage report, a Qlty coverage token, and the Qlty CLI.

You must instrument the mechanics over which to exercise your code base and collect coverage metrics, such as with automated testing for example, and provide the final coverage report to Qlty, that’s the entry port to integrate with Qlty.

General Procedure

The general procedure for uploading code coverage reports is as follows:

1

Locate the coverage report generated by your code coverage tool. This report is typically part of the output from running automated tests, such as unit tests, configured to monitor code coverage.

2

Retrieve your Qlty coverage token from your project’s settings page under the Code Coverage section.

3

Use the Qlty CLI to upload your report:

$curl https://qlty.sh | sh
  • Upload your code coverage report
$QLTY_COVERAGE_TOKEN=your_token qlty publish PATH

Checkout the coverage publish command’s documentation for more information.

The Qlty CLI integrates seamlessly into your CI/CD pipeline. For templates and examples to help you get started with popular CI/CD providers, refer to the Continuous Integration (CI) section.

How to treat coverage tokens

It’s advisable to treat coverage tokens as sensitive information whenever possible. Some CI/CD suites, like GitHub Actions, offer mechanisms to store sensitive information, such as secrets, which can be used in your workflows. This is an excellent place to store your Qlty code coverage token.

We recommend treating this token as sensitive information. However, if the token is leaked, the potential damage is minimal, as the primary risk is someone uploading dummy coverage reports to your projects.

Rotate a coverage token

In your project’s settings under the Code Coverage page, you will find an option to refresh your coverage token. This will provide you with a fresh token and invalidate the previous one.

Supported Languages and Formats

LanguageFormat / Coverage Tool
RubySimplecov
JavaScriptLcov (generated by Istanbul)
PHPClover
GoClover
JavaCobertura and Jacoco
SwiftCobertura (using native Xcode coverage and slather)
KotlinCobertura and Jacoco
QltyThis is Qlty’s generic coverage format

Report Format Specification

When uploading a coverage report, the format is typically inferred from the file extension or contents. However, you can explicitly specify the format using the --report-format option, for acceptable values query the Qlty CLI with qlty publish --help.

Path fixing

For Qlty to accurately match a coverage report file or any referenced file in your Git repository, the paths must align. Specifically, the path specified in your coverage report for a file must be relative to your Git project’s root. However, this alignment is not always present, as some tools generate coverage reports with absolute paths or may include additional content in the paths.

To resolve this issue, both qlty coverage publish and qlty coverage transform commands support the following flags:

  • --add-prefix: The prefix to add to file paths in coverage payloads.
  • --strip-prefix: The prefix to remove from absolute paths in coverage payloads. This makes paths relative to the project root, typically the directory where tests were run. Defaults to the current working directory.

Advanced configurations

Parallel testing

The Qlty CLI can merge parallelized coverage reports into a single cohesive report for submission to Qlty.

Configure your CI to store partial results from each parallel run. It’s recommended to sync and fetch files from S3 or use a temporary folder within your CI build. Then, utilize the qlty coverage publish to merge and upload the combined coverage report.

  1. Execute your test suite and store partial results in a shared and accessible location.
  2. Use qlty coverage publish PATH_TO_REPORT_1 PATH_TO_REPORT_2 PATH_TO_REPORT_3 to merge and upload a single cohesive coverage report to Qlty.

If you need to individually transform coverage reports, you can use qlty coverage transform.

  1. Run qlty coverage transform on reports from each parallel instance and store the results in a shared accessible location.
  2. Collect each transformed report and upload them using qlty coverage publish coverage_report_1.jsonl coverage_report_2.jsonl.

A successful qlty coverage transform PATH run produces a single file named coverage.jsonl by default, following the JSON Lines text format.

Multiple Test Suites

The procedure described above for merging multiple coverage reports from parallel testing can also be applied to combine coverage reports from multiple test suites. For example, you can merge coverage metrics from your unit test suite with those generated by your integration test suite.

We have a private beta for a feature which supports tracking code coverage from different test suites as independent metrics. If you are interested in this, please contact us.

See also