Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.qlty.sh/llms.txt

Use this file to discover all available pages before exploring further.

​
What is Matrix Testing?

Matrix testing runs a test suite across multiple combinations of languages, platforms, and frameworks to ensure compatibility and reliability across different environments. For example, a Ruby application might run tests across multiple Ruby and Rails versions:
Ruby VersionRails 6.1Rails 7.0Rails 7.1
Ruby 2.7✓✓✗
Ruby 3.0✓✓✓
Ruby 3.1✓✓✓
Ruby 3.2✗✓✓
Each cell represents a unique test configuration that produces its own coverage report.

​
Code Coverage for Matrix Testing

​
Option 1: Representative Coverage (Primary Configuration Only)

Choose a single matrix configuration as representative of your entire test suite and only report coverage for that configuration. This approach prioritizes speed over comprehensiveness by focusing on your primary supported environment. Example: 
name: Ruby and Rails Matrix Test with Representive Coverage

on: [push, pull_request]

jobs:
    test:
        runs-on: ubuntu-latest
        strategy:
            matrix:
                ruby: [3.1, 3.2, 3.3]
                rails: [6.1, 7.0, 7.1]
        env:
            RAILS_VERSION: ${{ matrix.rails }}
        steps:
            - name: Set up Ruby
              uses: ruby/setup-ruby@v1
              with:
                  ruby-version: ${{ matrix.ruby }}
                  bundler-cache: true

            - name: Checkout code
              uses: actions/checkout@v4

            - name: Install dependencies
              run: |
                  gem install bundler
                  bundle install --jobs 4 --retry 3

            - name: Run tests
              run: bundle exec rake

            - name: Upload coverage (only for primary configuration)
              if: ${{ matrix.ruby == '3.1' && matrix.rails == '7.0' }}
              uses: qltysh/qlty-action/coverage@v2
              with:
                  command: publish
                  token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
                  files: coverage/coverage.lcov
Advantages:
  • Fast feedback - reports immediately when primary configuration completes
  • Simple setup with minimal configuration
  • Focuses on most important environment
Disadvantages:
  • Not comprehensive across all supported configurations
  • May miss coverage gaps specific to certain environments
  • Assumes representative configuration covers edge cases

​
Option 2: Treat as Single Test Suite

When matrix testing focuses on compatibility rather than coverage analysis, you can merge all matrix cells into a single comprehensive report. This approach treats the matrix as “complete” only when all configurations have finished running. Example: 
name: Ruby and Rails Matrix Test with Final Coverage Complete Call

on: [push, pull_request]

jobs:
    test:
        runs-on: ubuntu-latest
        strategy:
            matrix:
                ruby: [3.1, 3.2, 3.3]
                rails: [6.1, 7.0, 7.1]
        env:
            RAILS_VERSION: ${{ matrix.rails }}
        steps:
            - name: Set up Ruby
              uses: ruby/setup-ruby@v1
              with:
                  ruby-version: ${{ matrix.ruby }}
                  bundler-cache: true

            - name: Checkout code
              uses: actions/checkout@v4

            - name: Install dependencies
              run: |
                  gem install bundler
                  bundle install --jobs 4 --retry 3

            - name: Run tests
              run: bundle exec rake

            - uses: qltysh/qlty-action/coverage@v2
              with:
                  command: publish
                  token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
                  files: coverage/coverage.lcov
                  incomplete: true

    final-step:
        needs: test
        runs-on: ubuntu-latest
        if: ${{ success() }} # Only run if all previous jobs succeeded
        steps:
            - uses: qltysh/qlty-action/coverage@v2
              with:
                  command: complete
                  token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
Advantages:
  • Provides comprehensive coverage across all configurations
  • Single unified coverage report
  • Best representation of overall test coverage
Disadvantages:
  • Must wait for complete matrix to complete before reporting
  • May require a final workflow step to signal completion to Qlty
  • Slower feedback loop

​
Option 3: Independent Suites with Tags

Track coverage separately for each matrix configuration using unique tags. This approach enables comparison of test coverage across different environments and platforms. Example: 
name: Ruby and Rails Matrix Test with Representive Coverage

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        ruby: [3.1, 3.2, 3.3]
        rails: [6.1, 7.0, 7.1]
    env:
      RAILS_VERSION: ${{ matrix.rails }}
    steps:
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: ${{ matrix.ruby }}
          bundler-cache: true

      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3

      - name: Run tests
        run: bundle exec rake

      - name: Upload coverage (only for primary configuration)
        uses: qltysh/qlty-action/coverage@v2
        with:
          command: publish
          token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
          files: coverage/coverage.lcov
          tag: "ruby-${{ matrix.ruby }}-rails-${{ matrix.rails }}"
This creates separate coverage reports for each combination:
  • ruby-3.0-rails-6.1
  • ruby-3.1-rails-7.0
  • ruby-3.2-rails-7.1
  • etc.
Advantages:
  • Compare coverage differences across configurations
  • Identify environment-specific test gaps
  • Track coverage trends for each supported platform
Disadvantages:
  • Creates a noisy user experience because every tag combination reports its own gate and summary comment. A 3x3 matrix would report 9 commit statuses.
  • Qlty does not currently rollup complete coverage across tags on commit statuses nor summary comments

​
Frequently Asked Questions

Q: What happens if I accidentally upload the same report for every matrix configuration using the default (empty) tag? A: Qlty automatically sums all coverage reports with the same tag. While the final result would be equivalent to a single comprehensive report, the user experience during execution could be confusing. As each matrix cell completes and uploads its report, the total coverage and diff coverage percentages may fluctuate until all configurations finish running. Q: Which approach should I choose for my project? A: Consider your priorities:
  • Fast feedback: Choose Option 1 (representative cell)
  • Comprehensive coverage: Choose Option 2 (single test suite)
  • Platform comparison: Choose Option 3 (independent suites with tags)
Choose one approach consistently to avoid confusing coverage comparisons between different reporting strategies.