Create a Multi-Job Pipeline

Create a Pipeline

1. Click New Pipeline... and wait for TeamCity to use the OAuth connection you provided to scan a VCS and display the list of available repositories.

2. Click any repository from the list. In this tutorial, I use a fork of this GitHub repository. This repository contains a Gradle project that builds a Spring Boot application, and a Dockerfile for making a Docker image.

3. Gradle recommends running builds with the help of the Gradle Wrapper — a script that allows builds to utilize the specific version of Gradle and, if no such version is present, download it before the target build starts. Enable the Use Gradle Wrapper option in the Step settings section to tell TeamCity it should look for the Wrapper inside the repository.

4. Enter build/libs/todo.jar to the Publish files field to tell TeamCity it should publish a resulting .jar file when the Gradle Step finishes.

5. Save and run your Pipeline and ensure it finishes successfully. Note that published files are now available from the Artifacts tab of the build results page.

   

Add a Dependent Job

Now that a Pipeline contains a Job that produces a required binary, we can add another Job that utilizes this file to build a Docker image.

1. On the main Pipeline page, click Add... to create a new Job.

2. In the Dependencies section, tick a checkbox next to the name of your first Job. By doing so, you specify that Job #2 depends on Job #1 and cannot run until the first Job finishes. The visual graph next to the settings pane confirms this link by placing both Jobs along the same timeline.

3. Our second Job must create a Docker image from the .jar file produced and shared by the first Job. To do this, choose "Use All Files" in the Dependencies section.

   

4. Add a "Script" Step and specify the following script body: docker build --pull --file ./docker/Dockerfile --tag myusername/myrepositoryname:mycustomtag .. This script runs the docker build command with the following parameters:

   • --pull — Attempts to pull a newer version of the image.

   • --file — The path to the Dockerfile.

   • --tag — The name for your build Docker image.

   • . — Sets the agent checkout directory as the context for the docker build command.

5. Add another line to your script: docker push myusername/myrepositoryname:mycustomtag. This command uploads your newly built image to the DockerHub repository.

6. If you run the Pipeline now, it will fail since the command added in the previous Step cannot upload images anonymously. You need to specify credentials for a user with the "Write" repository permission. To do this, expand the Integrations section, click Add | Docker Repository, and enter valid user credentials.

7. Save the Pipeline and run it. Check your DockerHub repository to ensure the second Job uploaded a Docker image.

Add Simultaneously Running Jobs

Our sample project has two modules with tests, each with its own set of Gradle instructions inside build.gradle files. With Pipelines, you can add different Jobs that run these test sets simultaneously.

1. Add a new Job that depends on the Job that builds a Docker image (select a corresponding Job in the Dependencies section). Since this Job does not require any files produced by other Jobs, you do not need to specify file paths in the Use files... dialog.

2. Add a new Gradle Step to this new Job.

3. All new Steps have "repository root" next to their Working directory setting. A working directory is a path on a build agent's local storage that stores files fetched from a remote repository. Whenever you run a Step, it uses files from this local repository copy. If you need to point a Step to the specific directory instead of the entire repository root folder, specify a required directory path (relative to the root directory). In our case, type test1 in the Working directory field.

4. If you want to manually select a Gradle build file, type a path to this file in the corresponding field. Note that the path must be relative to the current working directory. For instance, since our working directory is “test1”, you can type build.gradle and TeamCity will use the correct test1\build.gradle file rather than a similar file from the project root folder that Job #1 utilizes.

5. Check Use Gradle Wrapper to let TeamCity look for Wrapper files inside the "test1" directory.

6. Repeat steps 1 to 5 to create another Job, but use test2 instead of test1 when setting the Working directory. This directory contains its own build.gradle and Gradle Wrapper files, so other settings should remain the same.

7. You now have two Jobs that can start as soon as the Job that builds a Docker image finishes. The visual graph splits the sequence of Jobs into separate branches to illustrate this setup. Save and run your Pipeline to ensure both testing Jobs run their Steps simultaneously.

YAML Configuration

The final Pipeline should have the following YAML configuration. You can go to Pipeline settings, click the Preview YAML button in the top right corner, and compare your current settings with this reference configuration to check whether some of your settings are missing or have different values.