Migrating from Code Climate Quality

Migrating Code Coverage from Quality to Qlty Cloud

This guide is for users who have previously set up code coverage reporting to Code Climate Quality and are now migrating to Qlty Cloud.

Advantages of Qlty Cloud

The new Qlty Cloud app offers many advantages over Code Climate Quality for code coverage. To help make coverage setup easier, we offer four new features:

  1. Support for OpenID Connect (OIDC) with GitHub Actions for tokenless coverage uploads
  2. Workspace-level coverage tokens
  3. Reusable GitHub Action and CircleCI Orb tools
  4. Server-side coverage report merging

Once coverage is set up, Qlty Cloud supports all of the code coverage reporting features of Code Climate Quality like:

  • Coverage Tags for tracking data for e.g. tracking unit test coverage separately from integration test coverage
  • In-app views for reviewing coverage on Pull Request diffs (No broweser extension required)
  • Additional flexibility when configuring coverage gates (pass/fail commit statuses)

Prerequisites

  1. A Code Climate Quality account with repositories where you’ve previously configured coverage reporting
  2. A GitHub user which is an Administrator of your GitHub organization
  3. Ability to modify and run CI scripts for the relevant repositories

Process

Step 1. Determine an authentication approach

Qlty Cloud supports three ways to authenticate code coverage uploads from your CI system:

  1. OpenID Connect (OIDC) with GitHub Actions
  2. Workspace-level coverage tokens
  3. Project-level coverage tokens

If you are using GitHub Actions and are able to use OIDC, we recommend that approach as it is the most secure and requires no management of long-lived coverage tokens.

If you are not using GitHub Actions, or are unable to use OIDC, you can use a workspace-level coverage token. This is a long-lived token that is scoped to your entire organization and can be used for all repositories within that organization.

For situations were neight of those options are available, you can use a project-level coverage token. This is a long-lived token that is scoped to a single repository and can be used for that repository only.

Obtain a coverage token from Qlty Cloud

If you are not using OIDC, you will need to obtain a coverage token from Qlty Cloud.

Qlty Cloud uses GitHub for both authentication and authorization checks. Therefore, any user with administrator permissions on a GitHub organization is automatically able to access the corresponding workspace on Qlty Cloud.

  • Login to Qlty via GitHub OAuth (as a GitHub Admin)
  • Navigate to Workspace Settings > Code Coverage (e.g. https://qlty.sh/gh/org-name/settings/coverage)
  • Copy the workspace’s coverage token

Step 2. Store the coverage token as a CI secret (if not using OIDC)

If you are using OIDC, you can skip this step.

A coverage token only provides the ability to upload code coverage data to Qlty Cloud and does not provide access to read any data or administer any settings. Even still, it should be treated as a credential and stored securely within your CI system, as explained here

The specifics for how to store a secret at the organization level and make it available to all repositories vary by provider:

Step 3. Update CI workflow scripts

This guide assumes that your CI scripts are already generating code coverage data from the execution of your automated unit tests. The method to generate this data and the output format varies by programming language and testing tools.

Once you have code coverage data being generated in one of our supported formats, continue with the steps below. Generally, the process is to add an additional step after the code coverage data has been generated and written to disk on the CI runner.

Reusable GitHub Action

Add a step to run the qltysh/qlty-action/coverage reusable GitHub Action:

1# ...
2permissions:
3  contents: read
4  id-token: write # IMPORTANT
5
6jobs:
7  build:
8    # ...
9    steps:
10      # ...
11
12      - uses: qltysh/qlty-action/coverage@main
13        with:
14          oidc: true
15          files: COVERAGE_FILE_PATH # e.g. coverage/lcov.info

If you are using a workspace or project coverage token instead of OIDC, remove the oidc: true line and add coverage-token: ${{ secrets.QLTY_COVERAGE_TOKEN }} to the with block:

1# ...
2jobs:
3  build:
4    # ...
5    steps:
6      # ...
7
8      - uses: qltysh/qlty-action/coverage@main
9        with:
10          coverage-token: ${{ secrets.QLTY_COVERAGE_TOKEN }}
11          files: COVERAGE_FILE_PATH # e.g. coverage/lcov.info

CircleCI Orb

We provide a CircleCI Orb for uploading coverage reports. To use it, add the following to your .circleci/config.yml file:

1# ...
2jobs:
3  run_tests_and_publish_coverage:
4    # ...
5    steps:
6      # ...
7      - qlty/coverage_publish:
8          files: coverage/lcov.info

Learn more

Other CI providers

With other CI providers, you will install the Qlty CLI and run the qlty coverage upload command:

$# Assumes $QLTY_COVERAGE_TOKEN is set in the environment
>curl https://qlty.sh | sh
>qlty coverage publish COVERAGE_FILE_PATH

The Qlty CLI is able to auto-detect workflow-related metadata from the environment variables set by your CI provider on the following platforms:

  • Buildkite
  • Travis CI
  • CodeFresh
  • Semaphore

If you are using another CI provider not in this list, please contact us and we can add support for your system.

Step 4. Validate coverage uploads from a branch

After updating the CI script, trigger a build (of the branch including your updated script) and ensure it completes successfully. Check the output for the upload step and verify that no errors are occurring on the CI side during the upload process.

  • On Qlty Cloud, navigate to your Project Settings > Code Coverage page (example: https://qlty.sh/gh/org-name/projects/repo-name/settings/coverage).
  • Your uploaded coverage report should appear in the table at the bottom of the page
  • Click on the commit SHA to view your report details
  • Verify that the report details match your expectations (commit SHA, branch name, etc.)

Step 5. Merge your CI script changes into your main branch

Now that the coverage uploads are happening successfully for your branch, merge the branch into your default branch (e.g. main) and run a build.

Ensure that the build completes successfully (with no errors from the coverage upload step), and validate the coverage upload in your Project Settings.

Once Qlty receives a valid coverage report for your default branch, your coverage info will be displayed in your project’s Overview page. You’ll also see file-level and directory-level coverage on your project’s Issues tab.

Step 6. Repeat for each repository

Steps 3 through 5 need to be repeated for each repository that you are migrating to Qlty Cloud. We have found that with a good prompt for an AI coding assistant, the process can be completed quickly.

Optional: Enable coverage commit statuses from Qlty Cloud

  • Enable commit statuses (Qlty Coverage and Qlty Diff Coverage) within your Project’s Commit Statuses settings: (https://qlty.sh/gh/org-name/projects/repo-name/settings/gates)
  • Adjust coverage check settings (including new configurations for minimum diff size and total coverage decrease)

Optional: Remove coverage uploads to Code Climate Quality

  • Once you’ve setup coverage reporting (and enabled coverage commit statuses) as desired, remove references to the old Code Climate test reporter within your CI script.

Resources

Get Help