Concepts

Coverage Comparisons

Qlty provides powerful tools for comparing code coverage between different branches, commits, and pull requests. These comparisons help you understand how code changes affect your test coverage.

Types of Coverage Comparisons

Branch Comparisons

Compare coverage between any two branches to understand coverage differences:

  • Main vs. Feature Branch: See how coverage in a feature branch differs from the main branch
  • Between Feature Branches: Compare coverage between different implementations
  • Historical Branches: Compare current coverage with previous release branches

Commit Comparisons

Compare coverage between specific commits:

  • Before and After a Change: See how a particular change affected coverage
  • Between Release Tags: Compare coverage between different versions
  • Historical Commit Trends: Track coverage evolution over a series of commits

Pull Request Comparisons

Pull request comparisons happen automatically when you submit a PR:

  • Base vs. Head: Compare coverage between the base branch and the PR branch
  • Diff Coverage: See coverage specifically for the changed lines
  • Coverage Impact: Understand how merging would affect overall project coverage

How Coverage Comparisons Work

Comparison Methodology

When comparing coverage, Qlty:

  1. Analyzes the coverage reports for both comparison points
  2. Normalizes the data to ensure fair comparison (accounting for file renames, moves, etc.)
  3. Calculates metrics for total coverage, file-level coverage, and line-level coverage
  4. Identifies significant changes and highlights them
  5. Presents the results in a visual, actionable format

Common Metrics in Comparisons

  • Total Coverage Change: How overall coverage percentage changed
  • File Coverage Changes: Which files gained or lost coverage
  • Newly Covered/Uncovered Lines: Which specific lines gained or lost coverage
  • Diff Coverage: For changed files, what percentage of changes are covered

Using Coverage Comparisons

In Pull Requests

Pull request comparisons help reviewers understand the coverage impact of changes:

  1. Open a pull request on GitHub
  2. Qlty automatically adds a coverage comment showing comparison results
  3. Review the coverage data to identify potential gaps
  4. Address coverage issues before merging

In the Qlty Dashboard

For more detailed comparisons, use the Qlty dashboard:

  1. Navigate to your project in Qlty
  2. Go to the “Coverage” tab
  3. Select “Compare Coverage”
  4. Choose the branches or commits to compare
  5. View the detailed comparison report

The comparison report includes:

  • Side-by-side coverage metrics
  • File-by-file coverage changes
  • Visual coverage diffs
  • Actionable recommendations

From the Command Line

You can also perform comparisons using the Qlty CLI:

$# Compare coverage between two branches
>qlty coverage compare --base main --head feature/new-feature
>
># Compare coverage between two specific commits
>qlty coverage compare --base-commit abc123 --head-commit def456
>
># Export comparison as a report
>qlty coverage compare --base main --head feature/new-feature --output report.json

Comparison Features

Side-by-Side File View

The side-by-side file view shows coverage highlighting for both versions:

  • Green: Covered lines
  • Red: Uncovered lines
  • Gray: Not executable
  • Yellow: Partially covered (for branch coverage)

Lines that gained or lost coverage are specially highlighted to draw attention to changes.

Coverage Trend Graph

For branch and commit comparisons, a coverage trend graph shows how coverage has evolved between the two points:

  • Line graph of overall coverage
  • Markers for significant changes
  • Trendline showing the general direction

Impact Analysis

The impact analysis shows how the changes affect different parts of your codebase:

  • Most improved files
  • Files with decreased coverage
  • Files with consistent coverage
  • New files without adequate coverage

Coverage Change Summary

A statistical summary helps you quickly understand the comparison:

  • Number of files with improved coverage
  • Number of files with decreased coverage
  • Net impact on overall coverage percentage
  • Coverage for new and modified code

Best Practices

When to Use Different Comparison Types

  • Branch Comparisons: For understanding big-picture coverage differences between development streams
  • Commit Comparisons: For detailed analysis of how specific changes affected coverage
  • PR Comparisons: For everyday code review workflows

Interpreting Comparison Results

  • Look beyond the numbers: A small percentage change might hide significant file-level changes
  • Focus on important files: Pay special attention to core business logic and critical infrastructure
  • Consider context: Some coverage decreases might be acceptable if they’re accompanied by other quality improvements
  • Check for coverage gaps in new code: Ensuring new code is well-tested is often more important than fixing old coverage issues

Acting on Comparison Results

  • Add tests for newly uncovered lines: Prioritize writing tests for high-risk areas
  • Investigate unexpected coverage losses: These might indicate problems with test execution
  • Document deliberate coverage decisions: If you decide not to test certain code, document the reason
  • Set up alerts for significant coverage regressions: Be notified if coverage drops below certain thresholds

Troubleshooting

Common Issues

  • Missing comparison data: Ensure coverage reports exist for both comparison points
  • Misleading comparisons due to file moves: Use the “track moved files” option for accurate comparisons
  • Unexpected coverage changes: Check for indirect coverage impacts (changes in one file affecting coverage of another)

Addressing False Positives/Negatives

  • File renames causing false negatives: Use the path mapping feature to track renamed files
  • Generated code skewing comparisons: Configure path exclusions to ignore generated files
  • Test environment differences: Ensure tests are run consistently in the same environment

See Also