Unexpected Code Coverage Changes

When working with code coverage, you might occasionally see unexpected changes in coverage metrics. This guide helps you understand why these changes occur and how to investigate them.

There are two high level reasons why your coverage might change unexpectedly:

1. Execution Path Changes
2. Incomplete or Missing Coverage Data

Execution Path Changes

Qlty is designed to trust code coverage data exactly as it is generated by coverage tools like Cobertura, LCOV, SimpleCov, etc. These coverage tools often reflect reality: coverage changed because tests executed code differently.

The question that naturally follows is: what caused these changes?

While tests may not fail, coverage can change for all the same reasons that tests can occassionally fail: code changes, dependency changes, non-determinism, and other environment changes can cause your code to execute slightly differently on subsequent runs, even for the same commit.

Indirect Code Paths

• Function Call Removal: If File A stops calling a function in File B, File B’s coverage might decrease
• Conditional Logic Changes: Changing a condition in one file might prevent code in another file from executing
• Error Handling Changes: Changes to how errors are thrown or caught can affect which code paths are exercised

Example:

1
// Before: UserService.js
2
async function getUser(id) {
3
  return await UserRepository.findById(id); // UserRepository.findById is called every time
4
}
5
6
// After: UserService.js
7
async function getUser(id) {
8
  if (!id) return null; // Now UserRepository.findById is only called when id exists
9
  return await UserRepository.findById(id);
10
}

This change would reduce coverage in UserRepository.js even though it wasn’t modified.

Dependency Changes

Any sizeable software project relies on dependencies, some explicit and direct, and some less obvious and less direct. Here’s an (incomplete) list of dependencies that can cause execution changes and therefore coverage changes:

• Explicit dependency changes (e.g. Bumping a package from version 1.2.3 to 1.3.4)
• Floating dependencies (e.g. your code runs on the latest version of nodejs, and the latest version of nodejs changes)
• Time: a time change can cause a test to follow a different execution path (even for the same commit)
• Network connectivity
• Other environmental changes

Non-Determinism and Flaky Coverage

As noted, unexpected coverage changes can be viewed through the same lens as engineers often view unexpected test failures. Just like a test can be flaky and fail occassionally, coverage can change for similar reasons.

Incomplete or missing coverage reports

Test optimization tools

Test optimization tools sometimes intentionally or unintentionally (buggy) run less than 100% of tests. If the test optimization is new, this behavior would consistently result in Qlty reporting coverage drops.

Overwritten or overlooked coverage data files

Test suites commonly generate multiple coverage data files. Qlty must receive all coverage data files in order to accurately report coverage. While this can be straight-forward — a file was not sent that should have — there are a few less straight-forward ways this can happen:

• A build parallelizes their test suite using multiple threads or processes, but each unit of parallelization overwrites one another’s coverage data file, clobbering or corrupting coverage data
• A build parallelizes their test suite using multiple machines, where each machine generates unique coverage data files. Then, either some machines’s coverage data files are missed or the files, when collected together from different machines, because similarly named, overwrite one another.

Review our Coverage Merging documentation to ensure you’re sending coverage data correctly.

See “Investigating raw coverage data” below for more information on how to verify Qlty has received all coverage reports.

Investigating unexpected coverage changes

When you encounter unexpected coverage changes, you can debug the issue by comparing the actual coverage data between your base and head commits. This helps verify that the coverage difference is real and not a reporting error.

Investigating Qlty Cloud data

1. Obtain commit SHAs: Obtain the commit SHAs for both the branch head and merge base commits from Git or GitHub.

2. Navigate to coverage uploads: In your Qlty project, go to Project Settings > Code Coverage and search by commit SHA to find the correct coverage uploads for both commits. (NB: You can also search by branch)

3. Filter by file path: Use the on-page search to filter the coverage uploads by the specific file path that’s showing unexpected changes.

4. Compare coverage metrics: Confirm that the Covered Lines and/or Missed Lines values are differ between the two uploads.

Investigating raw coverage data

To further investigate the coverage difference, you can download and compare the raw coverage data files:

1. Obtain commit SHAs: Obtain the commit SHAs for both the branch head and merge base commits from Git or GitHub.

2. Navigate to coverage uploads: In your Qlty project, go to Project Settings > Code Coverage and search by commit SHA to find the correct coverage uploads for both commits. (NB: You can also search by branch)

3. Click “Download Zip”

2. Examine the coverage data fils: Extract the ZIP files and open the coverage data files in the raw_files/ folder. Search for the file path and compare the relevant sections between the two files.

3. Verify the difference: Each coverage file format is different, but you can often identify discrepencies by reading the data in a text editor.

This process helps confirm whether the indirect coverage change is internally consistent (caused by actual test execution differences) or if there might be an issue with how coverage is being generated or reported in your CI/CD pipeline.

After completing your investigation, you’ll encounter one of two outcomes:

• If the coverage data file shows different coverage values, this confirms the unexpected coverage change is real and legitimate. The coverage difference exists in your actual test execution between commits, meaning Qlty is correctly reporting what your coverage tools generated.

• If the coverage data shows identical coverage for both the base and head commits, this suggests an inconsistency in Qlty’s processing. If this is the case, feel free to report it to Qlty support with your findings, including the commit SHAs and file paths.