Coverage Metrics

Qlty provides several different code coverage metrics to give you a comprehensive understanding of your test coverage. This page explains each metric and how to interpret them.

Core Coverage Metrics

Total Coverage

Total Coverage measures the percentage of executable lines in your entire codebase that are covered by tests. It’s calculated using the formula:

Total Coverage = Covered Lines / Executable Lines

Lines that are not executable (comments, blank lines, etc.) are excluded from this calculation.

When to use it:

  • As a high-level indicator of overall project health
  • For tracking long-term trends in test coverage
  • For setting minimum coverage thresholds for the entire codebase

Example: A project with 1,500 covered lines out of 2,000 executable lines has a Total Coverage of 75%.

Diff Coverage

Diff Coverage focuses only on the lines changed in a pull request or commit. It shows the percentage of new or modified executable lines that are covered by tests.

Diff Coverage = Covered Changed Lines / Executable Changed Lines

When to use it:

  • For evaluating the test coverage of specific changes
  • As a quality gate for pull requests
  • To ensure new code maintains or improves coverage

Example: A pull request that adds 50 lines of code with 45 of them covered by tests has a Diff Coverage of 90%.

Total Coverage Delta

Total Coverage Delta shows how the Total Coverage has changed between two points, typically between the current branch and the base branch in a pull request.

Total Coverage Delta = New Total Coverage - Previous Total Coverage

When to use it:

  • To understand the impact of changes on overall project coverage
  • To prevent merging changes that significantly decrease coverage
  • To track progress toward coverage goals

Example: If the base branch has 75% coverage and your changes result in 76% coverage, the Total Coverage Delta is +1%.

File-level Metrics

File Coverage

File Coverage shows the percentage of executable lines in a specific file that are covered by tests.

File Coverage = Covered Lines in File / Executable Lines in File

When to use it:

  • To identify specific files that need more tests
  • To prioritize test writing efforts
  • To ensure critical files have high coverage

Example: A file with 90 covered lines out of 100 executable lines has a File Coverage of 90%.

Directory Coverage

Directory Coverage shows the average coverage across all files in a directory.

Directory Coverage = Sum of Covered Lines in Directory / Sum of Executable Lines in Directory

When to use it:

  • To understand coverage at the module or component level
  • To identify areas of the codebase that need more testing
  • To set coverage goals for specific components

Example: A directory containing files with a total of 500 covered lines out of 800 executable lines has a Directory Coverage of 62.5%.

Coverage Ratings

Qlty assigns letter grades to your coverage percentages to provide an easy-to-understand quality indicator.

Coverage PercentageQlty Coverage Rating
>= 90%A
>= 80%B
>= 70%C
>= 60%D
< 60%F

These ratings appear on your project dashboard and in coverage badges.

Advanced Metrics

Coverage by Tag

When using Coverage Tags, Qlty tracks separate coverage metrics for each tag, allowing you to see coverage from different test types.

Example:

  • Unit test coverage: A (95%)
  • Integration test coverage: B (85%)
  • E2E test coverage: C (75%)

Coverage Over Time

Qlty tracks how your coverage metrics change over time, allowing you to see trends:

  • Coverage Trend: Graph showing total coverage over time
  • Weekly Change: How coverage has changed in the last week
  • Monthly Change: How coverage has changed in the last month
  • Quarterly Change: How coverage has changed in the last quarter

Coverage Hotspots

Coverage Hotspots identify files with low coverage that are frequently changed, highlighting areas where adding tests would have the most impact.

The hotspot score is calculated based on:

  • How frequently the file changes
  • How low the file’s coverage is
  • How important the file is (based on interconnectedness)

Interpreting Coverage Metrics

What’s a “Good” Coverage Percentage?

The ideal coverage percentage depends on your project’s context:

  • 90%+ (A): Excellent for critical libraries, core business logic, and open-source projects
  • 80-90% (B): Very good for most application code
  • 70-80% (C): Acceptable for many projects
  • 60-70% (D): May indicate testing gaps
  • Below 60% (F): Usually indicates insufficient testing

When to Focus on Different Metrics

  • Total Coverage: For overall project health and long-term trends
  • Diff Coverage: For ensuring new code is well-tested
  • File Coverage: For identifying specific areas that need tests
  • Directory Coverage: For understanding module-level testing quality
  • Coverage Deltas: For preventing coverage regression

Limitations of Coverage Metrics

While coverage metrics are valuable, they don’t tell the whole story:

  • High coverage doesn’t guarantee high-quality tests
  • Some code may be inherently hard to test (e.g., UI, error handling)
  • Not all code is equally important to test
  • Coverage doesn’t measure test effectiveness at finding bugs

Use coverage as one of several metrics to evaluate your testing strategy, not as the only goal.

Best Practices

Setting Realistic Goals

  • Start with achievable coverage goals and gradually increase them
  • Set higher coverage requirements for critical code
  • Focus on Diff Coverage for new code rather than trying to immediately increase Total Coverage

Effective Coverage Standards

Typical coverage standards might include:

  • All new code must have at least 80% Diff Coverage
  • Critical modules must maintain at least 90% coverage
  • Total Coverage should not decrease

Using Coverage Gates

Configure Coverage Gates to enforce your coverage standards:

  • Diff Coverage Gate: Ensures new code meets coverage standards
  • Total Coverage Gate: Prevents significant coverage decreases
  • Custom Gates: For specific requirements like “100% coverage for security code”

See Also