Reference

Troubleshooting

This guide helps you troubleshoot common issues with code coverage in Qlty.

Upload Issues

Coverage Report Not Being Detected

Symptoms:

  • “Unsupported format” errors
  • “Unable to parse coverage report” messages
  • No coverage data appears in Qlty after upload

Potential Causes:

  • Unsupported coverage format
  • Corrupt or incomplete coverage report
  • Incorrect file path in upload command

Solutions:

  1. Verify your coverage format is supported
  2. Check that the coverage report file exists and is complete
  3. Try using the --format flag to explicitly specify the format: 
    $qlty coverage publish --format=lcov path/to/coverage.info
  4. Examine the report file for valid content
  5. Generate a fresh coverage report and try again

Authentication Failures

Symptoms:

  • “Authentication failed” errors
  • “Invalid token” messages
  • “Permission denied” errors

Potential Causes:

  • Incorrect coverage token
  • Expired or rotated token
  • Missing environment variable
  • Using a token with incorrect scope

Solutions:

  1. Verify your token is correct in your CI settings
  2. Check environment variable names (should be QLTY_COVERAGE_TOKEN)
  3. Regenerate your token in Qlty and update your CI configuration
  4. Ensure you’re using the token for the correct workspace or project
  5. If using OIDC, verify the permissions configuration in your GitHub workflow

Path Mapping Issues

Symptoms:

  • Coverage appears but doesn’t match files in your repository
  • “File not found” errors when viewing coverage
  • Coverage shown for files that don’t exist in your repo

Potential Causes:

  • Coverage report contains absolute paths
  • Path structure differs between build environment and repository
  • Generated files with different paths

Solutions:

  1. See Path Fixing for detailed solutions.

S3 Upload Failures (403 Errors)

Symptoms:

  • HTTP 403 Forbidden errors during coverage upload
  • “Access denied” messages when uploading to S3
  • Coverage upload fails with permission errors

Potential Causes:

  • Restrictive AWS VPC endpoint policies that block cross-account S3 operations
  • VPC endpoint policy doesn’t allow s3:PutObject action to other AWS accounts

Solutions:

We are aware of 1 situation which can cause our coverage uploader to fail to upload with a 403 error: if you have a restrictive AWS VPC endpoint policy which does not allow the S3:PutObject action to another AWS account.

In these cases, you’ll need to open up your VPC to allow this action. Here are examples of how to configure this:

1{
2  effect = "Allow"
3  actions = ["s3:PutObject"]
4  resources = ["*"]
5  principals = {
6    type = "AWS"
7    identifiers = ["*"]
8  }
9  conditions = [
10    {
11      test = "StringEquals"
12      variable = "aws:PrincipalAccount"
13      values = ["QLTY_AWS_ACCOUNT_ID"]
14    },
15    {
16      test = "StringEquals"
17      variable = "aws:ResourceAccount"
18      values = ["QLTY_AWS_ACCOUNT_ID"]
19    }
20  ]
21}

Reach out to customer support to obtain Qlty’s AWS account id.

Missing Metadata

Symptoms:

  • Coverage is uploaded but not associated with the correct branch
  • Coverage doesn’t appear in pull requests
  • Coverage data shows with incorrect commit information

Potential Causes:

  • CI system metadata not detected
  • Shallow git clone in CI
  • Custom CI setup not recognized by Qlty

Solutions:

  1. Manually specify metadata with override flags: 
    $qlty coverage publish --override-branch=main --override-commit-sha=abcdef123 coverage.xml
  2. Ensure your CI environment provides standard environment variables
  3. Use a deeper git clone in CI (or fetch git metadata)
  4. Check that the git repository is available in the CI environment

Viewing Issues

Coverage Not Showing in GitHub

Symptoms:

  • No coverage status checks on pull requests
  • Missing coverage comments on pull requests
  • Coverage data visible in Qlty but not in GitHub

Potential Causes:

  • GitHub app permissions issues
  • Coverage gates not enabled
  • PR comments not configured
  • Timing issues with CI and status reporting

Solutions:

  1. Verify GitHub integration settings in project settings
  2. Check that coverage gates are enabled in quality gates settings
  3. Ensure PR comments are enabled
  4. Check GitHub app permissions in your organization settings
  5. Try manually uploading coverage for the PR branch

Inconsistent Coverage Reports

Symptoms:

  • Different coverage percentages in different views
  • Coverage fluctuates between builds without code changes
  • Coverage shown in CI logs differs from what appears in Qlty

Potential Causes:

  • Different coverage criteria (line vs branch vs statement)
  • Test flakiness or non-deterministic tests
  • Cache issues in tests
  • Different coverage tool configurations

Solutions:

  1. Standardize coverage configuration across environments
  2. Fix flaky tests that produce inconsistent results
  3. Use the same coverage tool and settings consistently
  4. Implement deterministic test ordering
  5. Clear caches between test runs

Missing Coverage for Some Files

Symptoms:

  • Some files show no coverage data
  • Certain directories are missing from coverage reports
  • Coverage appears for some languages but not others

Potential Causes:

  • Include/exclude patterns filtering out files
  • Multi-language projects with incomplete coverage setup
  • Files not being exercised by tests
  • Generated code or third-party code inclusion/exclusion

Solutions:

  1. Check coverage tool configuration for include/exclude patterns
  2. Verify that all relevant files are included in coverage report generation
  3. Set up coverage for each language in multi-language projects
  4. Use multiple coverage reports for different parts of the codebase
  5. Configure path exclusions appropriately in Qlty

Quality Gate Issues

Failing Coverage Gates

Symptoms:

  • Pull requests failing due to coverage checks
  • “Coverage decreased” warnings even with improved test coverage
  • Diff coverage failing despite high coverage in changed files

Potential Causes:

  • Thresholds set too high for your project’s current state
  • Coverage decreasing in files not directly modified
  • Base branch coverage calculation issues
  • Coverage calculation includes excluded files

Solutions:

  1. Adjust coverage gate thresholds to realistic levels
  2. Investigate unexpected coverage changes
  3. Temporarily disable gates while building up coverage
  4. Use path exclusions to ignore files that shouldn’t affect gates
  5. Focus on diff coverage rather than total coverage for incremental improvement

Coverage Badges Not Updating

Symptoms:

  • Outdated coverage percentage in badges
  • Coverage badge shows “unknown” or “N/A”
  • Badge doesn’t reflect recent coverage improvements

Potential Causes:

  • Badge caching
  • No coverage data for default branch
  • Badge URL issues

Solutions:

  1. Ensure coverage is being uploaded for your default branch
  2. Check badge URLs and project settings
  3. Manually refresh cached badges
  4. Verify that the badge is pointing to the correct project and metric
  5. Wait for cache expiration (typically a few hours)

Merging

Coverage Data Missing

Presenting Problem

  • Qlty reports less coverage for a source file than expected. For example, a line is covered according to the raw coverage data, but Qlty shows the line as not covered
  • Qlty does not show coverage data for a file

Diagnosis and Resolutions

  1. Ensure that Qlty has received all partial reports (uploads).

    The best diagnostic tool is the list of coverage uploads browsable on qlty.sh at Project Settings => Code Coverage. Search for the commit sha, pull request (e.g. refs/pull/5), or branch (e.g. refs/heads/branchName). You should see N “incomplete” reports representing the N uploads Qlty received as well as 1 “complete” report, representing a completion of the report. If you see less than you expect, check that:

    • Every upload attempt succeeded. Did the CLI report “Upload successful” at the end of its output? If not, what error occurred?
    • Every coverage report attempted an upload. Providing a unique name with every coverage report (with --name) can help pinpoint a missing coverage report. (The name will be show in the coverage upload detail page)
    • Each upload is associated with the same expected tag (or no tag)

  2. Avoid data conflicts between upload reports

    Every line in a coverage report is exclusively either “covered”, “uncovered”, or “omitted” (An “omitted” line is a non-executable line which cannot be covered by a test - a code comment is the canonical example).

    Qlty expects that every coverage upload (for the same commit) agrees on which lines are executable (and therefore which lines are omitted), but there is no enforcement of this requirement. Therefore, it’s possible for Qlty to receive two different uploads for the same commit and each to report conflicting coverage information: one report may say a line is omitted and another that the line is covered. While this is technically irreconcilable data, Qlty, to maintain consistency in reporting, will always treat a line reported as omitted to be omitted, regardless of whether the line is also reported as covered or uncovered.

Merging Coverage Reports

Symptoms:

  • Coverage from parallel tests not combining correctly
  • Missing coverage data from certain test suites
  • Merged coverage showing lower than expected percentages

Potential Causes:

  • Incorrect merging configuration
  • Incompatible coverage report formats
  • Issues with multi-part uploads

Solutions:

  1. Review coverage merging documentation
  2. For client-side merging, ensure all reports use the same format
  3. For server-side merging, verify the --total-parts-count configuration
  4. Check that all parts are successfully uploaded
  5. Try converting all reports to a common format before merging

Coverage Tag Issues

Symptoms:

  • Tagged coverage not appearing separately
  • Incorrect coverage shown for specific tags
  • Tags not filtering correctly in UI

Potential Causes:

  • Incorrect tag specification during upload
  • Mixing tagged and untagged uploads
  • UI filtering confusion

Solutions:

  1. Verify tag naming in upload commands: 
    $qlty coverage publish --tag=unit coverage.xml
  2. Ensure consistent tag usage across uploads
  3. Check tag configuration in project settings
  4. Use different tags for different test types (unit, integration, etc.)

Large Repository Issues

Symptoms:

  • Coverage uploads timing out
  • Very slow coverage rendering in UI
  • Missing coverage for some files in large repositories

Potential Causes:

  • Coverage reports too large
  • Too many files included in coverage
  • Performance limits in browser extensions

Solutions:

  1. Use path exclusions to limit coverage to relevant code
  2. Split large repositories into smaller projects
  3. Exclude generated code, vendor code, or test code from coverage
  4. Use more specific coverage focus for very large repositories
  5. Contact Qlty support for help with large-scale deployments

Getting Additional Help

If you’ve tried the solutions above and are still experiencing issues:

  1. Contact Support: Reach out to Qlty support
  2. Join Discord: Ask for help in our Discord community
  3. GitHub Issues: Submit a bug report on our GitHub issue tracker

When asking for help, include:

  • The exact command you’re running
  • Any error messages (with sensitive information redacted)
  • Your CI provider and configuration
  • The coverage tool and language you’re using
  • Steps you’ve already taken to troubleshoot