Coverage Metrics
Qlty provides several different code coverage metrics to give you a comprehensive understanding of your test coverage. This page explains each metric and how to interpret them.
Core Coverage Metrics
Total Coverage
Total Coverage measures the percentage of executable lines in your entire codebase that are covered by tests. It’s calculated using the formula:
Lines that are not executable (comments, blank lines, etc.) are excluded from this calculation.
When to use it:
- As a high-level indicator of overall project health
- For tracking long-term trends in test coverage
- For setting minimum coverage thresholds for the entire codebase
Example: A project with 1,500 covered lines out of 2,000 executable lines has a Total Coverage of 75%.
Diff Coverage
Diff Coverage focuses only on the lines changed in a pull request or commit. It shows the percentage of new or modified executable lines that are covered by tests.
When to use it:
- For evaluating the test coverage of specific changes
- As a quality gate for pull requests
- To ensure new code maintains or improves coverage
Example: A pull request that adds 50 lines of code with 45 of them covered by tests has a Diff Coverage of 90%.
Total Coverage Delta
Total Coverage Delta shows how the Total Coverage has changed between two points, typically between the current branch and the base branch in a pull request.
When to use it:
- To understand the impact of changes on overall project coverage
- To prevent merging changes that significantly decrease coverage
- To track progress toward coverage goals
Example: If the base branch has 75% coverage and your changes result in 76% coverage, the Total Coverage Delta is +1%.
File-level Metrics
File Coverage
File Coverage shows the percentage of executable lines in a specific file that are covered by tests.
When to use it:
- To identify specific files that need more tests
- To prioritize test writing efforts
- To ensure critical files have high coverage
Example: A file with 90 covered lines out of 100 executable lines has a File Coverage of 90%.
Directory Coverage
Directory Coverage shows the average coverage across all files in a directory.
When to use it:
- To understand coverage at the module or component level
- To identify areas of the codebase that need more testing
- To set coverage goals for specific components
Example: A directory containing files with a total of 500 covered lines out of 800 executable lines has a Directory Coverage of 62.5%.
Coverage Ratings
Qlty assigns letter grades to your coverage percentages to provide an easy-to-understand quality indicator.
These ratings appear on your project dashboard and in coverage badges.
Advanced Metrics
Coverage by Tag
When using Coverage Tags, Qlty tracks separate coverage metrics for each tag, allowing you to see coverage from different test types.
Example:
- Unit test coverage: A (95%)
- Integration test coverage: B (85%)
- E2E test coverage: C (75%)
Coverage Over Time
Qlty tracks how your coverage metrics change over time, allowing you to see trends:
- Coverage Trend: Graph showing total coverage over time
- Weekly Change: How coverage has changed in the last week
- Monthly Change: How coverage has changed in the last month
- Quarterly Change: How coverage has changed in the last quarter
Coverage Hotspots
Coverage Hotspots identify files with low coverage that are frequently changed, highlighting areas where adding tests would have the most impact.
The hotspot score is calculated based on:
- How frequently the file changes
- How low the file’s coverage is
- How important the file is (based on interconnectedness)
Interpreting Coverage Metrics
What’s a “Good” Coverage Percentage?
The ideal coverage percentage depends on your project’s context:
- 90%+ (A): Excellent for critical libraries, core business logic, and open-source projects
- 80-90% (B): Very good for most application code
- 70-80% (C): Acceptable for many projects
- 60-70% (D): May indicate testing gaps
- Below 60% (F): Usually indicates insufficient testing
When to Focus on Different Metrics
- Total Coverage: For overall project health and long-term trends
- Diff Coverage: For ensuring new code is well-tested
- File Coverage: For identifying specific areas that need tests
- Directory Coverage: For understanding module-level testing quality
- Coverage Deltas: For preventing coverage regression
Limitations of Coverage Metrics
While coverage metrics are valuable, they don’t tell the whole story:
- High coverage doesn’t guarantee high-quality tests
- Some code may be inherently hard to test (e.g., UI, error handling)
- Not all code is equally important to test
- Coverage doesn’t measure test effectiveness at finding bugs
Use coverage as one of several metrics to evaluate your testing strategy, not as the only goal.
Best Practices
Setting Realistic Goals
- Start with achievable coverage goals and gradually increase them
- Set higher coverage requirements for critical code
- Focus on Diff Coverage for new code rather than trying to immediately increase Total Coverage
Effective Coverage Standards
Typical coverage standards might include:
- All new code must have at least 80% Diff Coverage
- Critical modules must maintain at least 90% coverage
- Total Coverage should not decrease
Using Coverage Gates
Configure Coverage Gates to enforce your coverage standards:
- Diff Coverage Gate: Ensures new code meets coverage standards
- Total Coverage Gate: Prevents significant coverage decreases
- Custom Gates: For specific requirements like “100% coverage for security code”
See Also
- Coverage Comparisons - How to compare coverage between branches
- Coverage Gates - Using coverage as a quality gate
- Coverage Trends - Tracking coverage changes over time