Coverage Merging

Many builds generate multiple code coverage reports. Some examples include:

• Builds involving multiple programming languages, where each language requires different coverage instrumentation
• Parallelized builds where each parallel unit (thread, process, virtual machine etc.) may generate its own coverage report
• Different test suites (unit, integration, end-to-end) generating separate reports

Qlty supports merging these separate reports into a single comprehensive report using either Client-Side or Server-Side merging.

• If builds occur on a single machine (or multiple machines with shared storage), client-side merging is often the simplest approach.
• If builds are spread across multiple machines and disks, server-side merging is likely the easiest solution.

Client-side merging

Client-side merging uses the Qlty CLI to combine multiple coverage reports into a single report. That single report is then uploaded to Qlty Cloud for processing.

This is accomplished by passing the locations of your raw coverage reports (as paths or globs) to the qlty coverage publish subcommand:

1
- uses: qltysh/qlty-action/coverage@v1
2
  with:
3
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
4
    files: |
5
        coverage/coverage1.lcov
6
        coverage/coverage2.lcov
7
        coverage/coverage3.lcov

Or using glob patterns:

1
- uses: qltysh/qlty-action/coverage@v1
2
  with:
3
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
4
    files: coverage/*.lcov

The CLI transforms each of these reports into the Qlty format, then publishes a single coverage report to Qlty Cloud.

Client-side merging is ideal when all of your coverage data files are on a single machine.

Server-side merging

With server-side merging, a build publishes a series of partial reports to Qlty; when Qlty Cloud knows it’s received all parts (as determined by the below options), it aggregates and reports coverage for that commit SHA.

Server-side merging is ideal when:

• Your tests run on multiple machines without shared storage
• You want to avoid collecting all reports in a single location
• Each report file may require unique path fixing

Qlty provides two strategies for server-side merging:

coverage complete

coverage complete style coverage aggregation is our preferred server side aggregation option. To use it, upload all coverage parts, identifying each part as “incomplete”, and then send a complete signal to Qlty:

1
- uses: qltysh/qlty-action/coverage@v1
2
  with:
3
    command: publish
4
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
5
    files: coverage/coverage1.lcov
6
    incomplete: true
7
8
- uses: qltysh/qlty-action/coverage@v1
9
  with:
10
    command: publish
11
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
12
    files: coverage/coverage2.lcov
13
    incomplete: true
14
15
- uses: qltysh/qlty-action/coverage@v1
16
  with:
17
    command: complete
18
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}

When Qlty receives the completion signal, it will finalize the coverage report, merging all the received parts.

Pros:

• Coverage is only merged when a build finishes successfully. (For this reason, this style of merging is preferred)
• The number of parts being sent does not need to be known in advance

Cons:

• Requires configuring an additional final step to be run only when the build finishes successfully

Late-arriving incomplete parts

If incomplete parts are received after a qlty coverage complete command has been issued, these parts will be incorporated into the final report as shown on the coverage tab of the PR on qlty.sh. However, any pull request statuses previously sent and/or summary comments will not be updated to reflect this new coverage data.

To update the pull request with the latest coverage information after late-arriving parts, issue another qlty coverage complete command. This will trigger an update to both the PR statuses and summary comment to reflect the current merged coverage report.

total-parts-count

This approach works by specifying a --total-parts-count=NUM argument to the qlty coverage publish subcommand when publishing a coverage part. If, for example, your build consists of 2 parallelized builds, Qlty Cloud will wait for 2 coverage publish commands:

1
- uses: qltysh/qlty-action/coverage@v1
2
  with:
3
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
4
    files: coverage/coverage1.lcov
5
    total-parts-count: 2
6
7
- uses: qltysh/qlty-action/coverage@v1
8
  with:
9
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
10
    files: coverage/coverage2.lcov
11
    total-parts-count: 2

When using total-parts-count, Qlty only ever merges reports for the same build, as determined by the buildId inferred or provided explicitly to the coverage publish command (with the --override-build-id flag).

Pros:

• Your CI does not need to be configured with an additional final step as is the case with the coverage complete style option.

Cons:

• Because Qlty does not receive a “complete” signal indicating the build has finished, this solution has edge cases which could cause reports to be merged prematurely or not merged. If this is a concern, prefer the complete style merging option.
• Qlty cannot merge parts across different builds to reduce the above noted edge cases

Client and server-side merging together

Client-side and server-side merging can also be used together: each machine itself can combine multiple coverage reports using client-side merging, and then post that combined report to Qlty as part of server-side merging.

The total parts count always equals the number of total publish commands executed, regardless of the number of paths passed to the subcommand. If 2 machines perform client-side aggregation of N paths, qlty coverage publish should specify “2” as the total parts count.

1
# Job 1: Combine and upload unit test reports
2
- uses: qltysh/qlty-action/coverage@v1
3
  with:
4
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
5
    total-parts-count: 2
6
    files: |
7
        unit1.lcov
8
        unit2.lcov
9
        unit3.lcov
10
11
# Job 2: Combine and upload integration test reports
12
- uses: qltysh/qlty-action/coverage@v1
13
  with:
14
    token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
15
    total-parts-count: 2
16
    files: |
17
        integration1.lcov
18
        integration2.lcov

Using merging with coverage tags

When using coverage tags with report merging, please refer to the Coverage Tags documentation for detailed information.

The key points to remember:

• Each qlty coverage publish command applies to a single tag (or no tag)
• Merging always refers to combining reports for the same tag, not across different tags
• Server-side merging requires tracking completion separately for each tag

Common Scenarios

Build matrix

In GitHub Actions or other CI systems that support matrix builds:

1
jobs:
2
  test:
3
    strategy:
4
      matrix:
5
        node-version: [14.x, 16.x, 18.x]
6
        os: [ubuntu-latest, windows-latest]
7
8
    runs-on: ${{ matrix.os }}
9
10
    steps:
11
      # ... run tests and generate coverage ...
12
13
      - name: Upload coverage
14
        uses: qltysh/qlty-action/coverage@v1
15
        with:
16
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
17
          files: coverage/lcov.info
18
          # 3 node versions × 2 operating systems = 6 parts
19
          total-parts-count: 6

Multiple languages

For a project with both JavaScript and Python code:

1
jobs:
2
  test-javascript:
3
    runs-on: ubuntu-latest
4
    steps:
5
      # ... run JS tests and generate coverage ...
6
7
      - name: Upload JS coverage
8
        uses: qltysh/qlty-action/coverage@v1
9
        with:
10
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
11
          files: coverage/lcov.info
12
          total-parts-count: 2 # JavaScript + Python = 2 parts
13
14
  test-python:
15
    runs-on: ubuntu-latest
16
    steps:
17
      # ... run Python tests and generate coverage ...
18
19
      - name: Upload Python coverage
20
        uses: qltysh/qlty-action/coverage@v1
21
        with:
22
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
23
          files: coverage.xml
24
          total-parts-count: 2 # JavaScript + Python = 2 parts

Verifying merging

After publishing your coverage reports, you can verify that merging was successful:

1. Navigate to your project in Qlty Cloud
2. Go to Project Settings → Code Coverage
3. Filter the table below “Code Coverage Setup” by commit
4. You should see N + 1 reports:
   • N individual parts representing the reports you sent
   • 1 final merged report representing the combined coverage

Troubleshooting

Missing parts

If your merged report doesn’t appear after several minutes:

For Total Parts Count approach:

1. Check that all parts were successfully uploaded
2. Verify that the --total-parts-count value matches the actual number of uploads
3. Ensure all parts were uploaded for the same commit SHA and branch
4. Check that all parts used the same tag (if using tags)

Completion Signal Issues

If you’re using the incomplete flag approach and experiencing issues:

1. Ensure that all parts were uploaded with the --incomplete flag (otherwise, qlty coverage publish assumes by default that the report is complete)
2. Verify the completion command was executed for each tag you’re tracking
3. Check that both upload and completion commands are using the same credentials and workspace
4. Verify all coverage parts were uploaded with the same commit SHA and branch

Partial merging

If some files are missing from your merged coverage:

1. Verify that all coverage reports were valid and complete
2. Check for path differences between reports
3. Ensure all reports use compatible formats
4. Consider using path fixing options if paths don’t match

See Also

• Coverage Tags - Track different types of tests separately
• CI Integration - Automate coverage reporting in your CI pipeline
• Path Fixing - Fix path discrepancies in coverage reports