dotCover 2023.3 Help

Set up Coverage on JetBrains TeamCity

Normally, a continuous integration (CI) build for a .NET solution consists of a series of tasks that involve compiling and running tests. In terms of JetBrains TeamCity CI server, these tasks are called build steps. As TeamCity comes bundled with dotCover command-line tool, you can collect code coverage data by just enabling the corresponding build step option. TeamCity-collected coverage results can be viewed as a report inside TeamCity or downloaded as a coverage snapshot and viewed inside Visual Studio.

For example, we have a simple solution that consists of two projects: one stands for the application and the other one for the test assembly. Our task is to set up continuous integration build that compiles the application, runs tests, and collects tests coverage statistics. Note that the workflow differs depending on the unit test runner you use:

MSTest/NUnit

The MSTest and NUnit test runners are provided by TeamCity out of the box, so, generally, all you need is to add a build step with the corresponding runner.

Set up coverage analysis of MSTest/NUnit tests in TeamCity

  1. In TeamCity, create a new build project and setup the VCS root. Then add a new Build Step: Create a new project and a new build configuration inside the project.

  2. Add a first build step, that will build the solution and tests. In our particular example, we use the .NET runner (requires the following software to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, and so on.

    TeamCity build step

    Notice that all we are doing here is building the solution. No coverage settings yet.

  3. Add a build step that runs the tests. If you use MSTest tests, select the .NET runner with `vstest` as a Command. Our sample solution uses NUnit tests, therefore, we choose the NUnit runner. Here we also specify the NUnit version, .NET runtime parameters and path to the tests assembly.

    TeamCity build step: run tests
  4. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.

    TeamCity build step: coverage
  5. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MainTests assembly that contains running tests.

  6. Save the configuration and run it to see how it works.

  7. Once the build is finished, the Overview tab will contain a short report on code coverage.

    TeamCity run overview

    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods do the tests cover.

    TeamCity code coverage report

    Or even drill down into individual classes and examine the code coverage.

    TeamCity. Code coverage shown on source code

    Build artifacts now contain coverage files zipped up.

    TeamCity. Artifacts
  8. Note that in a real-life environment, you may have two different build configurations: one that builds binaries and the other one that collects code coverage. In that case, you should use snapshot dependency in order the second build configuration could find source files. Otherwise, you will see the Source code is not available message when trying to view source code.

    TeamCity. No source code

    To solve the issue, follow the instructions from the TeamCity documentation.

xUnit

The main problem with running and getting coverage of xUnit tests is that xUnit is not supported by TeamCity out of the box. This creates two issues that must be solved:

  1. xUnit packages must be provided to build a solution.

    Committing binary libraries to VCS is not a good practice, so, restoring the packages must be a separate build step before the compilation stage.

  2. xUnit test runner must be provided and run manually.

    The good news:

    • The runner is distributed as a NuGet package, so, it can be referenced right in your solution and then restored during a build (see a).

    • There are no additional actions required to get tests results as the runner provides the results in the format of TeamCity service messages.

Set up coverage analysis of xUnit tests in TeamCity

  1. In Visual Studio, open your solution. In the tests project, add a reference to the xunit.runner.console NuGet package.

    Installing XUnit console runner
  2. In TeamCity, create a new build project and setup the VCS root. Then add a new Build Step: Create a new project and a new build configuration inside the project.

  3. Make sure that TeamCity is configured as a NuGet server: in Administration | Tools find the NuGet.exe table and check whether it contains the required NuGet version. If there is no such a table, add the NuGet to TeamCity by clicking the Install Tool... button and following the required steps.

    TeamCity. XUnit NuGet server
  4. Add a build step, that restores NuGet packages referenced by the solution. Use NuGet Installer as the Runner type.

    TeamCity. NuGet installer step
  5. Add a first build step, that will build the solution and tests. In our particular example, we use the .NET runner (requires the following software to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, and so on.

    TeamCity build step

    Notice that all we are doing here is building the solution. No coverage settings yet.

  6. Add a build step that runs the tests:

    • As the xUnit runner will be executed as a simple .exe file, select the .NET runner.

    • In the Command, select <custom>.

    • As the runner is provided as a package inside the solution, you can specify its path relatively to the solution working directory, for example:

      %system.teamcity.build.workingDir%\SimpleTestsProject\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe
    • The Command line parameters field must contain the path to the compiled test assembly.

    TeamCity build steps. Run and cover
  7. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.

    TeamCity. XUnit coverage step
  8. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MainTests assembly that contains running tests and everything related to xUnit assemblies.

  9. Save the configuration and run it to see how it works.

  10. Once the build is finished, the Overview tab will contain a short report on test results and code coverage.

    TeamCity. XUnit run overview

    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods do the tests cover.

    TeamCity. XUnit coverage statistics

    Or even drill down into individual classes and examine the code coverage.

    TeamCity. Code coverage shown on source code

    Build artifacts now contain coverage files zipped up.

MSpec

MSpec is partly supported by TeamCity out of the box: it has a corresponding build step runner, but MSpec test runner is not in the bundle and must be installed to TeamCity agents manually. Fortunately, the runner is distributed as a NuGet package, so, it can be referenced right in your solution and then restored during a build.

Set up coverage analysis of MSpec tests in TeamCity

  1. In Visual Studio, open your solution. In the tests project, add a reference to the Machine.Specifications.Runner.Console NuGet package.

    Installing MSpec via NuGet
  2. In TeamCity, create a new build project and setup the VCS root. Then add a new Build Step: Create a new project and a new build configuration inside the project.

  3. Make sure that TeamCity is configured as a NuGet server: in Administration | Tools find the NuGet.exe table and check whether it contains the required NuGet version. If there is no such a table, add the NuGet to TeamCity by clicking the Install Tool... button and following the required steps.

    TeamCity. XUnit NuGet server
  4. Add a build step, that restores NuGet packages referenced by the solution. Use NuGet Installer as the Runner type.

    TeamCity. NuGet installer step
  5. Add a first build step, that will build the solution and tests. In our particular example, we use the .NET runner (requires the following software to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, and so on.

    TeamCity build step

    Notice that all we are doing here is building the solution. No coverage settings yet.

  6. Add a build step that runs the tests:

    • Select MSpec in the Runner type.

    • As the runner is provided as a package inside the solution, you can specify its path relatively to the solution working directory, for example:

      %system.teamcity.build.workingDir%\SimpleTestsProject\packages\Machine.Specifications.Runner.Console.0.9.3\tools\mspec-clr4.exe

      Make sure you use the runner version that corresponds the project's targeted .NET Framework version.

    • In Run tests from, specify the path to the compiled test assembly.

    • If needed, specify what specifications must be included/excluded in Include specifications and Exclude specifications.

    TeamCity. MSpec test runner step
  7. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.

    TeamCity. MSpec coverage step
  8. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MSpecTests assembly that contains running tests and everything related to MSpec assemblies.

  9. Save the configuration and run it to see how it works.

  10. Once the build is finished, the Overview tab will contain a short report on test results and code coverage.

    TeamCity. MSpec tests run overview

    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods were covered by tests.

    TeamCity. MSpec tests coverage statistics

    Or even drill down into individual classes and examine the code coverage.

    TeamCity. MSpec tests coverage shown on source code

    Build artifacts now contain coverage files zipped up.

Last modified: 01 December 2023