CI Integration / Uploader

Integrating code coverage reporting into your CI pipeline ensures that coverage data is consistently collected and reported for every build.

Qlty is compatible with every major CI provider.

Qlty offers CI-specific packages for GitHub Actions (GitHub Action) and CircleCI (CircleCI Orb), but works just as well without them.

Qlty provides environment detection for most CI providers, making integration simpler, but offers simple instructions for those without it.

CI Provider Integration

GitHub Actions

Integrate with Qlty using our GitHub Action.

While integration can be done without our GitHub Action, using it allows you to take advantage of OIDC support.

1
# .github/workflows/test.yml
2
name: Test and Coverage
3
4
on:
5
  push:
6
    branches: [ main ]
7
  pull_request:
8
    branches: [ main ]
9
10
permissions:
11
  contents: read
12
  id-token: write # Required for OIDC
13
14
jobs:
15
  test:
16
    runs-on: ubuntu-latest
17
    steps:
18
      - uses: actions/checkout@v3
19
      
20
      # Run your tests with coverage
21
      - name: Run tests
22
        run: npm test -- --coverage
23
      
24
      # Upload coverage to Qlty
25
      - uses: qltysh/qlty-action/coverage@v2
26
        with:
27
          oidc: true
28
          files: coverage/lcov.info

CircleCI

Qlty offers a CircleCI Orb for easy integration.

1
# Add the Qlty orb to your `.circleci/config.yml`:
2
version: 2.1
3
4
orbs:
5
  qlty-orb: qltysh/qlty-orb@0.0
6
7
jobs:
8
  test:
9
    docker:
10
      - image: cimg/node:16.13
11
    steps:
12
      # Checkout is required
13
      - checkout
14
      - run:
15
          name: Install dependencies
16
          command: npm ci
17
      - run:
18
          name: Run tests with coverage
19
          command: npm test
20
      - qlty-orb/coverage_publish:
21
          files: coverage/lcov.info
22
23
workflows:
24
  main:
25
    jobs:
26
      - test

Travis CI

Add Qlty coverage reporting to your .travis.yml:

1
language: node_js
2
node_js:
3
  - 16
4
5
script:
6
  - npm test -- --coverage
7
8
after_success:
9
  - curl https://qlty.sh | sh
10
  - qlty coverage publish coverage/lcov.info

Make sure to add your QLTY_COVERAGE_TOKEN in the Travis CI repository settings.

GitLab CI

Add coverage reporting to your .gitlab-ci.yml:

1
test:
2
  stage: test
3
  script:
4
    - npm ci
5
    - npm test -- --coverage
6
    - curl https://qlty.sh | sh
7
    - qlty coverage publish coverage/lcov.info
8
  artifacts:
9
    paths:
10
      - coverage/

Add your QLTY_COVERAGE_TOKEN as a CI/CD variable in your GitLab project settings.

Buildkite

Export your QLTY_COVERAGE_TOKEN as an environment variable. (See Buildkite’s documentation on managing secrets.

One option is to export QLTY_COVERAGE_TOKEN in an environment agent hook):

$
#!/usr/bin/env bash
>
set -e
>
>
export QLTY_COVERAGE_TOKEN="YOUR_COVERAGE_TOKEN"

Then update your pipeline.yaml appropriately. Here’s an example with Ruby test coverage:

1
steps:
2
  - label: ":ruby: Ruby 3.1 Tests"
3
    key: "ruby-tests"
4
    command: |
5
      bundle install
6
      bundle exec rspec --require spec_helper
7
    artifact_paths:
8
      - "coverage/coverage.json"
9
10
  - wait
11
12
  - label: ":chart_with_upwards_trend: Coverage Report"
13
    command: |
14
      # Download the coverage artifact
15
      buildkite-agent artifact download "coverage/coverage.json" .
16
17
      curl https://qlty.sh | sh
18
      qlty coverage publish coverage/coverage.json
19
    depends_on: "ruby-tests"

Other CI Providers

As long as your CI environment meets the system requirements to install the Qlt CLI, you can publish coverage.

Publishing coverage on any CI provider follows the same procedure:

1. Ensure a QLTY_COVERAGE_TOKEN is available in the environment
2. Install the Qlty CLI
3. Execute qlty coverage publish command with the correct arguments

For example:

$
# Assumes $QLTY_COVERAGE_TOKEN is set in the environment
>
curl https://qlty.sh | sh
>
qlty coverage publish coverage/lcov.info # Replace with path to your coverage data file(s)

The Qlty CLI coverage commands leverage as much information from its environment as possible to ease integration. This includes environment variables as well as git metadata from the current working directory. However, sometimes, such as when attempting to publish coverage from a CI provider not listed above, Qlty is unable to detect this information from its environment. In these cases, you will need to provide the missing metadata to qlty coverage publish as additional arguments. Because usually no more than 2 - 4 arguments are needed, we recommend a trial and error process, following the instructions provided by the command when it errors.

Below are some common scenarios you may run into.

Path Validation Failures

qlty coverage publish validates that the source files specified in your coverage data files match the source files in your repository. This ensures that we’re able to properly associate coverage with your source files in Qlty Cloud.

If we’re unable to validate that a majority (up to a configurable threshold) of the source files referenced exist in your repository, qlty coverage publish will error.

Usually these validate failures this can be addressed with the --add-prefix and/or --strip-prefix arguments. See Path Fixing for more information.

Commit SHA Missing

A commit sha is always required when sending coverage data to Qlty.sh. The commit sha can be manually specified with:

--override-commit-sha <COMMIT_SHA>

Reference Missing

When publishing coverage, Qlty associates the coverage data with a single reference. The reference can be one of:

• a branch
• a pull request
• a git tag

If not inferred, provide one of the following:

• pull request: --override-pr-number <PR_NUMBER>
• branch: --override-branch <BRANCH_NAME> (can be used alongside --override-pr-number)
• git tag: --override-git-tag <TAG_NAME> (cannot be combined with either --override-pr-number or --override-branch)

Commit Time Missing

A timestamp associated with the commit is also required. This can be specified with:

--override-commit-time <OVERRIDE_COMMIT_TIME>

The commit time can be a Unix timestamp (seconds since epoch) or a date in RFC3339/ISO8601 format.

Examples

$
QLTY_COVERAGE_TOKEN=QLTY_TOKEN qlty coverage publish --override-branch=main --override-commit-sha=e83c5163316f89bfbde7d9ab23ca2e25604af290 --override-commit=time=1704067200 lcov.info

Advanced CI Configuration

Parallelized Tests

When running tests in parallel, you’ll need to merge the coverage reports before uploading. Qlty supports both client-side and server-side merging.

Client-side Merging

Collect all coverage reports into a single location and upload them together:

$
qlty coverage publish report1.lcov report2.lcov report3.lcov

Server-side Merging

Upload each part separately and specify the total number of parts:

$
# Part 1
>
qlty coverage publish --total-parts-count=3 report1.lcov
>
>
# Part 2
>
qlty coverage publish --total-parts-count=3 report2.lcov
>
>
# Part 3
>
qlty coverage publish --total-parts-count=3 report3.lcov

Learn more about Coverage Merging.

Multiple Test Suites

If you have different types of tests (unit, integration, e2e), you can track them separately using coverage tags:

$
# Upload unit test coverage
>
qlty coverage publish --tag=unit unit-coverage.lcov
>
>
# Upload integration test coverage
>
qlty coverage publish --tag=integration integration-coverage.lcov

Learn more about Coverage Tags.

Environment Detection

What is environment detection?

Environment detection is Qlty CLI’s ability to infer, without user input, metadata about your build from CI provider-specific environment variables. For example, CircleCI exposes the commit being built in an environment variable CIRCLE_SHA1.

Without specific environment detection, you can still use our coverage publisher with a few extra command line arguments. See our documentation below on Other CI Providers.

Adding Environment Detection

If you want automatic environment detection:

1. Double check our Qlty repository for the most up to date list of supported CI providers.
2. Our coverage publisher is open source and you can add support by following the other provider examples in under an hour. (Provide an LLM with the other examples and a link to your CI provider’s list of available environment variables)

Troubleshooting

Common Issues

• Authentication failures: Check that your token is correctly set as an environment variable
• Report format not recognized: Make sure you’re using a supported coverage format
• Path mismatches: Use the --strip-prefix and --add-prefix options to fix file paths
• Missing metadata: Use --override-branch, --override-commit and --override-build-id if CI detection fails

See Also

• Coverage Merging - Combine multiple coverage reports
• Coverage Tags - Track different test suites separately