Reference

Indirect Code Coverage Changes

Qlty tells me there's coverage changes in that files I didn't change

While at first maybe not intuitive, it is possible, and in some cases very easy, for a change to one file to impact the code coverage metrics of another (untouched) file.

This can happen directly, and more obviously, by changing tests in such a way as to change the coverage of the files under tests, without a single change to the files under test.

However, it can also happen more indirectly. Consider a case where a function (under test) in File A makes a call to a function defined in File B. File B had 100% coverage … however, someone changes File A to remove the call to the function in File B: this reduces the coverage of File B indirectly.

There are also a number of other cases which can result in indirect coverage changes:

  • Incomplete or missing coverage reports
  • Dependency changes which change the execution path of tests
  • Tests that behave differently at different times of the day or in different environments

Investigating indirect coverage changes

When you encounter indirect coverage changes, you can debug the issue by comparing the actual coverage data between your base and head commits. This helps verify that the coverage difference is real and not a reporting error.

Qlty is designed to trust code coverage data exactly as it is generated by coverage tools like Cobertura, LCOV, SimpleCov, etc.

Therefore, the Qlty technical support team is unable to investigate indirect code coverage changes, as the causes are always unique to the codebase.

If you find a discrepancy between the raw coverage data generated by your coverage tool and the data reported by Qlty, please let us know.

Investigating Qlty Cloud data

  1. Obtain commit SHAs: Obtain the commit SHAs for both the branch head and merge base commits from Git or GitHub.

  2. Navigate to coverage uploads: In your Qlty project, go to Project Settings > Code Coverage and search by commit SHA to find the correct coverage uploads for both commits.

  3. Filter by file path: Use the on-page search to filter the coverage uploads by the specific file path that’s showing unexpected changes.

  4. Compare coverage metrics: Confirm that the Covered Lines and/or Missed Lines values are differ between the two uploads.

Investigating raw coverage data

To further investigate the coverage difference, you can download and compare the raw coverage data files:

  1. Download raw data: Download the ZIP files for both coverage uploads to examine the underlying data.

  2. Examine the coverage data fils: Extract the ZIP files and open the coverage data files in the raw_files/ folder. Search for the file path and compare the relevant sections between the two files.

  3. Verify the difference: Each coverage file format is different, but you can often identify discrepencies by reading the data in a text editor.

This process helps confirm whether the indirect coverage change is internally consistent (caused by actual test execution differences) or if there might be an issue with how coverage is being generated or reported in your CI/CD pipeline.

After completing your investigation, you’ll encounter one of two outcomes:

  • If the coverage data file shows different coverage values, this confirms the indirect coverage change is real and legitimate. The coverage difference exists in your actual test execution between commits, meaning Qlty is correctly reporting what your coverage tools generated.

  • If the coverage data shows identical coverage for both the base and head commits, this suggests an inconsistency in Qlty’s processing. If this is the case, feel free to report it to Qlty support with your findings, including the commit SHAs and file paths.