Quality gate

Quality gates are techniques that let you control your code quality and build software that meets your quality expectations. If a quality gate condition fails, Qodana terminates using exit code 255, which makes a CI/CD workflow or pipeline fail. For example, if a quality gate for a project is set to 10 problems and code coverage is set to 40%, a build workflow will fail if Qodana detects either an eleventh problem or code coverage below the 40% threshold.

The terms quality gate, threshold and fail threshold are used interchangeably.

You can configure a single quality gate for the total number of project problems, multiple quality gates for each separate problem severity, and code coverage thresholds as explained in the table below.

Quality gate type	Linter support	Availability under licenses
Total number of problems	All linters	Community, Ultimate, and Ultimate Plus
Severity-specific problems	All linters except Qodana Community for .NET	Community, Ultimate, and Ultimate Plus licenses depending on a linter
Code coverage threshold	Qodana for JVM, Qodana for JS, Qodana for PHP, Qodana for .NET, Qodana for Python, and Qodana for Go	Ultimate and Ultimate Plus

How it works

Qodana follows these rules:

Given the total number of project problems of A, when finding A problems in the project, the run will succeed. When finding A+1 problems, the run will fail.
Given the severity-specific number of problems of B, when finding B problems of a specific severity, the run will succeed. When finding B+1 problems, the run will fail.
Given a code coverage threshold of C, when C% of lines are covered, the run will succeed. If only C-1% are covered, the run will fail.

Total number of problems

You can configure a quality gate for the total number of project problems in all Qodana linters.

This is the basic qodana.yaml configuration supported by all linters:

failThreshold: <number>

Alternatively, all linters except Qodana Community for .NET support the severityThresholds.any option:

                    failureConditions:
                      severityThresholds:
                        any: <number-of-project-problems>
                

                    qodana scan \
                      -e QODANA_TOKEN="<cloud-project-token>" \
                      --fail-threshold <number>
                

In this command, the --fail-threshold <number> option configures the quality gate. The QODANA_TOKEN variable refers to the project token required by the Ultimate and Ultimate Plus linters.

                    docker run \
                      -v <source-directory>/:/data/project/  \
                      -e QODANA_TOKEN="<cloud-project-token>" \
                      jetbrains/qodana-<linter> \
                      --fail-threshold <number>
                

In this command, <source-directory> is the full local path to the project source code, and the --fail-threshold <number> option configures the quality gate. The QODANA_TOKEN variable refers to the project token, which is required by the Ultimate and Ultimate Plus linters.

You can configure GitHub to block the merging of pull requests if the quality gate has failed. To do this, you can create a branch protection rule as described below:

Create a new GitHub Actions workflow that invokes the Qodana scan action or open an existing one.
Set the workflow to run on pull_request events that target the main branch.
name: Qodana on: pull_request: branches: - main jobs: qodana: runs-on: ubuntu-latest permissions: contents: write pull-requests: write checks: write steps: - uses: actions/checkout@v3 with: ref: ${{ github.event.pull_request.head.sha }} # to check out the actual pull request commit, not the merge commit fetch-depth: 0 # a full history is required for pull request analysis - name: 'Qodana Scan' uses: JetBrains/qodana-action@v2024.1 env: QODANA_TOKEN: ${{ secrets.QODANA_TOKEN }}
Instead of main, you can specify your preferred branch here. The QODANA_TOKEN variable refers to the project token required by the Ultimate and Ultimate Plus linters.
Set the fail threshold (number) for the Qodana Action fail-threshold option.
Under your repository name, click Settings.
On the left menu, click Branches.
In the branch protection rules section, click Add rule.
Add main to Branch name pattern.
Select Require status checks to pass before merging.
Search for the Qodana status check, then check it.
Click Create.

To set up a quality gate in a Jenkins Pipeline, you can add the --fail-threshold <number> option to the steps block:

                    pipeline {
                       environment {
                          QODANA_TOKEN=credentials('qodana-token')
                       }
                       agent {
                          docker {
                             args '''
                                -v <path-to-project>:/data/project
                                --entrypoint=""
                             '''
                             image 'jetbrains/qodana-<linter>'
                          }
                       }
                       stages {
                          stage('Qodana') {
                             steps {
                                sh '''
                                   qodana \
                                   --fail-threshold <number>
                                '''
                             }
                          }
                       }
                    }
                

The QODANA_TOKEN variable in this snippet refers to the project token contained in the qodana-token credentials, which is required by the Ultimate and Ultimate Plus linters.

If you want to run a quality gate in a GitLab CI/CD pipeline, you can save this configuration to the .gitlab-ci.yml file:

                    qodana:
                     image:
                       name: jetbrains/qodana-<linter>
                       entrypoint: ['']
                    variables:
                       QODANA_TOKEN: $qodana_token
                    script:
                       - qodana --fail-threshold <number>
                    artifacts:
                      paths:
                        - qodana
                

In this sample, the script section specifies the --fail-threshold <number> option. The QODANA_TOKEN variable in this snippet refers to the project token, which is required by the Ultimate and Ultimate Plus linters.

Severity-specific problems

All linters except Qodana Community for .NET let you configure separate quality gates for each problem severity, and exceeding just one setting limitation will make the build fail, including the total number of problems.

Here's a qodana.yaml severity configuration:

            failureConditions:
              severityThresholds:
                any: <number> # Total problems
                critical: <number> # Critical and other severities
                high: <number>
                moderate: <number>
                low: <number>
                info: <number>
        

Code coverage threshold

You can configure the total and fresh code coverage thresholds for the Qodana for JVM, Qodana for JS, Qodana for PHP, Qodana for .NET, Qodana for Python, and Qodana for Go linters.

Here's a qodana.yaml configuration sample:

            failureConditions:
              testCoverageThresholds:
                fresh: <number> # Fresh code coverage
                total: <number> # Total code coverage
        

Last modified: 25 April 2024