dotCover 2018.2 Help

Set up Coverage on JetBrains TeamCity

Normally, a continuous integration (CI) build of a .NET solution consist of a series of tasks that involve compilation and running of tests. In terms of JetBrains TeamCity CI server, these tasks are called build steps. As TeamCity comes bundled with dotCover Console Runner, you can collect code coverage data by just enabling the corresponding build step option. TeamCity-collected coverage results can be viewed as a report inside TeamCity or downloaded as a coverage snapshot and viewed inside Visual Studio.

For example, we have a simple solution that consists of two projects: one stands for the application and the other one for the test assembly. Our task is to set up continuous integration build that compiles the application, runs tests, and collects tests coverage statistics. Note that the workflow differs depending on the unit test runner you use:

MSTest/NUnit

The MSTest and NUnit test runners are provided by TeamCity out of the box, so, generally, all you need is to add a build step with the corresponding runner.

To set up coverage analysis of MSTest/NUnit tests in TeamCity

  1. In TeamCity, create a new build project and setup the VCS root. We then add a new Build Step: Create a new project and a new build configuration inside the project.
  2. Add a first build step, that will build the application. In our particular example, we use the Visual Studio (sln) runner (requires Visual Studio to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, etc.
    tc build step1 compile
    Notice that all we are doing here is building the solution. No coverage settings yet.
  3. Add a build step that runs the tests. If you use MSTest tests, select Visual Studio Tests in Runner type. Our sample solution uses NUnit tests, therefore, we choose NUnit. Here we also specify the NUnit version, .NET runtime parameters and path to the tests assembly.
    tc build step2 run tests
  4. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.
    tc build step2 coverage
  5. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MainTests assembly that contains running tests.
  6. Save the configuration and run it to see how it works.
    tc run conf
  7. Once the build is finished, the Overview tab will contain a short report on code coverage.
    tc overview
    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods were covered by tests.
    tc code coverage
    Or even drill down into individual classes and examine the code coverage.
    tc source code
    Build artifacts now contain coverage files zipped up.
    tc artifacts
  8. Please note that in a real-life environment, you may have two different build configurations: one that builds binaries and the other one that collects code coverage. In that case, you should use snapshot dependency in order the second build configuration could find source files. Otherwise, you will see the Source code is not available message when trying to view source code.
    tc no code
    To solve the issue, please follow the instructions from the TeamCity documentation.

xUnit

The main problem with running and getting coverage of xUnit tests is that xUnit is not supported by TeamCity out of the box. This creates two issues that must be solved:

  • xUnit packages must be provided to build a solution.
    Committing binary libraries to VCS is not a good practice, so, restoring the packages must be a separate build step before the compilation stage.
  • xUnit test runner must be provided and run manually.
    The good news:
    • The runner is distributed as a NuGet package, so, it can be referenced right in your solution and then restored during a build (see a).
    • There are no additional actions required to get tests results as the runner provides the results in the format of TeamCity service messages.

To set up coverage analysis of xUnit tests in TeamCity

  1. In Visual Studio, open your solution. In the tests project, add a reference to the xunit.runner.console NuGet package.
    install xunit console
  2. In TeamCity, create a new build project and setup the VCS root. Then add a new Build Step: Create a new project and a new build configuration inside the project.
  3. Make sure, TeamCity is configured as a NuGet server: in Administration | Tools find the NuGet.exe table and check whether it contains the required NuGet version. If there is no such a table, add the NuGet to TeamCity by clicking the Install Tool... button and following the required steps.
    tc xunit nuget server
  4. Add a build step, that restores NuGet packages referenced by the solution. Use NuGet Installer as the Runner type.
    tc nuget installer step
  5. Add a build step, that will build the application. In our particular example, we use the Visual Studio (sln) runner (requires Visual Studio to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, etc.
    tc build step1 compile
    Notice that all we are doing here is building the solution. No coverage settings yet.
  6. Add a build step that runs the tests:
    • As the xUnit runner will be executed as a simple .exe file, select .NET Process in the Runner type.
    • As the runner is provided as a package inside the solution, you can specify its path relatively to the solution working directory, e.g.
      %system.teamcity.build.workingDir%SimpleTestsProject\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe
    • The Command line parameters field must contain the path to the compiled test assembly.
    tc xunit run cover step
  7. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.
    tc xunit coverage step
  8. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MainTests assembly that contains running tests and everything related to xUnit assemblies.
  9. Save the configuration and run it to see how it works.
  10. Once the build is finished, the Overview tab will contain a short report on test results and code coverage.
    tc xunit overview
    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods were covered by tests.
    tc xunit coverage stats
    Or even drill down into individual classes and examine the code coverage.
    tc xunit coverage source code
    Build artifacts now contain coverage files zipped up.

MSpec

MSpec is partly supported by TeamCity out of the box: it has a corresponding build step runner, but MSpec test runner is not in the bundle and must be installed to TeamCity agents manually. Fortunately, the runner is distributed as a NuGet package, so, it can be referenced right in your solution and then restored during a build.

To set up coverage analysis of MSpec tests in TeamCity

  1. In Visual Studio, open your solution. In the tests project, add a reference to the Machine.Specifications.Runner.Console NuGet package.
    tc mspec nuget
  2. In TeamCity, create a new build project and setup the VCS root. Then add a new Build Step: Create a new project and a new build configuration inside the project.
  3. Make sure, TeamCity is configured as a NuGet server: in Administration | Tools find the NuGet.exe table and check whether it contains the required NuGet version. If there is no such a table, add the NuGet to TeamCity by clicking the Install Tool... button and following the required steps.
    tc xunit nuget server
  4. Add a build step, that restores NuGet packages referenced by the solution. Use NuGet Installer as the Runner type.
    tc nuget installer step
  5. Add a build step, that will build the application. In our particular example, we use the Visual Studio (sln) runner (requires Visual Studio to be installed on a build agent), but you have a number of other options on how to build: directly run MSBuild, run batch file, etc.
    tc build step1 compile
    Notice that all we are doing here is building the solution. No coverage settings yet.
  6. Add a build step that runs the tests:
    • Select MSpec in the Runner type.
    • As the runner is provided as a package inside the solution, you can specify its path relatively to the solution working directory, e.g.
      %system.teamcity.build.workingDir%SimpleTestsProject\packages\Machine.Specifications.Runner.Console.0.9.3\tools\mspec-clr4.exe
      Make sure you use the runner version that corresponds the project's targeted .NET Framework version.
    • In Run tests from, specify the path to the compiled test assembly.
    • If needed, specify what specifications must be included/excluded in Include specifications and Exclude specifications.
    tc mspec runner step
  7. Now, it's time to set up coverage. In the .NET coverage tool, choose JetBrains dotCover.
    tc mspec coverage step
  8. In Assembly Filters, add the assemblies you want coverage for (just the name of the assembly) prefixing them with +: and filter out those you do not want coverage for with –:. In our example, we exclude the MSpecTests assembly that contains running tests and everything related to MSpec assemblies.
  9. Save the configuration and run it to see how it works.
  10. Once the build is finished, the Overview tab will contain a short report on test results and code coverage.
    tc mspec overview
    To see the details, switch to the Code Coverage tab. Here you can view the detailed statistics on how much of class/methods were covered by tests.
    tc mspec coverage stats
    Or even drill down into individual classes and examine the code coverage.
    tc mspec coverage source code
    Build artifacts now contain coverage files zipped up.
Last modified: 4 September 2018

See Also