Matrix Testing

What is Matrix Testing?

Matrix testing runs a test suite across multiple combinations of languages, platforms, and frameworks to ensure compatibility and reliability across different environments.

For example, a Ruby application might run tests across multiple Ruby and Rails versions:

Each cell represents a unique test configuration that produces its own coverage report.

Code Coverage for Matrix Testing

Option 1: Representative Coverage (Primary Configuration Only)

Choose a single matrix configuration as representative of your entire test suite and only report coverage for that configuration. This approach prioritizes speed over comprehensiveness by focusing on your primary supported environment.

Example:

1
name: Ruby and Rails Matrix Test with Representive Coverage
2
3
on: [push, pull_request]
4
5
jobs:
6
  test:
7
    runs-on: ubuntu-latest
8
    strategy:
9
      matrix:
10
        ruby: [3.1, 3.2, 3.3]
11
        rails: [6.1, 7.0, 7.1]
12
    env:
13
      RAILS_VERSION: ${{ matrix.rails }}
14
    steps:
15
      - name: Set up Ruby
16
        uses: ruby/setup-ruby@v1
17
        with:
18
          ruby-version: ${{ matrix.ruby }}
19
          bundler-cache: true
20
21
      - name: Checkout code
22
        uses: actions/checkout@v4
23
24
      - name: Install dependencies
25
        run: |
26
          gem install bundler
27
          bundle install --jobs 4 --retry 3
28
29
      - name: Run tests
30
        run: bundle exec rake
31
32
      - name: Upload coverage (only for primary configuration)
33
        if: ${{ matrix.ruby == '3.1' && matrix.rails == '7.0' }}
34
        uses: qltysh/qlty-action/coverage@v1
35
        with:
36
          command: publish
37
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
38
          files: coverage/coverage.lcov

Advantages:

• Fast feedback - reports immediately when primary configuration completes
• Simple setup with minimal configuration
• Focuses on most important environment

Disadvantages:

• Not comprehensive across all supported configurations
• May miss coverage gaps specific to certain environments
• Assumes representative configuration covers edge cases

Option 2: Treat as Single Test Suite

When matrix testing focuses on compatibility rather than coverage analysis, you can merge all matrix cells into a single comprehensive report. This approach treats the matrix as “complete” only when all configurations have finished running.

Example:

1
name: Ruby and Rails Matrix Test with Final Coverage Complete Call
2
3
on: [push, pull_request]
4
5
jobs:
6
  test:
7
    runs-on: ubuntu-latest
8
    strategy:
9
      matrix:
10
        ruby: [3.1, 3.2, 3.3]
11
        rails: [6.1, 7.0, 7.1]
12
    env:
13
      RAILS_VERSION: ${{ matrix.rails }}
14
    steps:
15
      - name: Set up Ruby
16
        uses: ruby/setup-ruby@v1
17
        with:
18
          ruby-version: ${{ matrix.ruby }}
19
          bundler-cache: true
20
21
      - name: Checkout code
22
        uses: actions/checkout@v4
23
24
      - name: Install dependencies
25
        run: |
26
          gem install bundler
27
          bundle install --jobs 4 --retry 3
28
29
      - name: Run tests
30
        run: bundle exec rake
31
32
      - uses: qltysh/qlty-action/coverage@v1
33
        with:
34
          command: publish
35
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
36
          files: coverage/coverage.lcov
37
          incomplete: true
38
39
  final-step:
40
    needs: test
41
    runs-on: ubuntu-latest
42
    if: ${{ success() }}  # Only run if all previous jobs succeeded
43
    steps:
44
      - uses: qltysh/qlty-action/coverage@v1
45
        with:
46
            command: complete
47
            token: ${{ secrets.QLTY_COVERAGE_TOKEN }}

Advantages:

• Provides comprehensive coverage across all configurations
• Single unified coverage report
• Best representation of overall test coverage

Disadvantages:

• Must wait for complete matrix to complete before reporting
• May require a final workflow step to signal completion to Qlty
• Slower feedback loop

Option 3: Independent Suites with Tags

Track coverage separately for each matrix configuration using unique tags. This approach enables comparison of test coverage across different environments and platforms.

Example:

1
```yaml
2
name: Ruby and Rails Matrix Test with Representive Coverage
3
4
on: [push, pull_request]
5
6
jobs:
7
  test:
8
    runs-on: ubuntu-latest
9
    strategy:
10
      matrix:
11
        ruby: [3.1, 3.2, 3.3]
12
        rails: [6.1, 7.0, 7.1]
13
    env:
14
      RAILS_VERSION: ${{ matrix.rails }}
15
    steps:
16
      - name: Set up Ruby
17
        uses: ruby/setup-ruby@v1
18
        with:
19
          ruby-version: ${{ matrix.ruby }}
20
          bundler-cache: true
21
22
      - name: Checkout code
23
        uses: actions/checkout@v4
24
25
      - name: Install dependencies
26
        run: |
27
          gem install bundler
28
          bundle install --jobs 4 --retry 3
29
30
      - name: Run tests
31
        run: bundle exec rake
32
33
      - name: Upload coverage (only for primary configuration)
34
        uses: qltysh/qlty-action/coverage@v1
35
        with:
36
          command: publish
37
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
38
          files: coverage/coverage.lcov
39
          tag: "ruby-${{ matrix.ruby }}-rails-${{ matrix.rails }}"

This creates separate coverage reports for each combination:

• ruby-3.0-rails-6.1
• ruby-3.1-rails-7.0
• ruby-3.2-rails-7.1
• etc.

Advantages:

• Compare coverage differences across configurations
• Identify environment-specific test gaps
• Track coverage trends for each supported platform

Disadvantages:

• Creates a noisy user experience because every tag combination reports its own gate and summary comment. A 3x3 matrix would report 9 commit statuses.
• Qlty does not currently rollup complete coverage across tags on commit statuses nor summary comments

Frequently Asked Questions

Q: What happens if I accidentally upload the same report for every matrix configuration using the default (empty) tag?

A: Qlty automatically sums all coverage reports with the same tag. While the final result would be equivalent to a single comprehensive report, the user experience during execution could be confusing. As each matrix cell completes and uploads its report, the total coverage and diff coverage percentages may fluctuate until all configurations finish running.

Q: Which approach should I choose for my project?

A: Consider your priorities:

• Fast feedback: Choose Option 1 (representative cell)
• Comprehensive coverage: Choose Option 2 (single test suite)
• Platform comparison: Choose Option 3 (independent suites with tags)

Choose one approach consistently to avoid confusing coverage comparisons between different reporting strategies.