Advanced Configuration

Carry Forward Tags

Carry forward tags allow Qlty to automatically reuse coverage data from previous commits when selective testing leaves coverage gaps, ensuring your coverage metrics remain complete and accurate even when only running a subset of tests.

Background

Many modern CI systems are configured to only run a subset of tests on each commit (also known as “selective testing”). One example of selective testing are “monorepos” where each folder represents a distinct component (or service). These repositories are often configured to limit test runs to only components that have changed.

With selective testing, coverage cannot be strictly compared across commits because those commits may have run different test suites, generating coverage reports that are unrelated. Because coverage data is associated with commits, selective testing creates unique challenges associated with per-commit coverage:

  • The latest commit (alone) for a project won’t have comprehensive coverage data
  • A pull request is a comparison across 2 commits, and these commits can have different coverage reports

Qlty solves this problem with the intelligent use of coverage tags: if a project defines a tag per component, Qlty “carries forward” coverage for a tag from a previous commit onto a more recent relevant commit. While this behavior is seamless and requires no setup other than defining tags, understanding Carry Forward Tags can help you comprehend how coverage calculations are performed in selective testing scenarios.

How Carry Forward Works

When Qlty processes coverage for a commit, it automatically fills in missing tag data by carrying forward the most recent coverage report for each tag.

For example, in the following diagram, while the most recent commit, at the top, does not have direct coverage report data, the coverage data associated with the coverage tags “component A” and “component B” will be carried forward.

This ensures that your dashboards, PR checks, and coverage overlays always show the most complete and up-to-date coverage information available, even if that coverage information spans multiple commits.

Best Practices

To make the most of the automatic carry forward behavior:

Consistent Tag Usage

Use consistent tag names across all your coverage uploads. This ensures proper tracking and carry-forward behavior.

Consistent Tag Names
$# Correct approach - consistent tag name
>qlty coverage publish --tag=unit unit-tests.lcov
>qlty coverage publish --tag=unit more-unit-tests.lcov
>
># Incorrect approach - inconsistent tag names
>
>qlty coverage publish --tag=unit unit-tests.lcov
>qlty coverage publish --tag=unit-tests more-unit-tests.lcov

Regular Complete Coverage

Periodically run and upload coverage for all tags, such as on your main branch or nightly builds. This ensures carried forward data doesn’t become too outdated.

Limitations

While carry forward is a valuable built-in behavior, it’s important to understand its limitations:

  1. Accuracy over time: Carried-forward data may become outdated if a component goes untested for a long period
  2. File-level granularity: Carry forward operates at the tag level, not at individual file or line level
  3. Branch integrity: Data is only carried forward within the same branch history
  4. Tag consistency: Carry-forward requires consistent usage of tags across commits — e.g. Carry forward will not work properly if the same tag reports information for a different set of tests each time

See Also