TeamCity On-Premises 2022.10 Help

NUnit Support in TeamCity

There are two most common methods to set up NUnit tests reporting in TeamCity:

  • Use the .NET build runner.

  • Use the NUnit build runner.

Besides, you can try alternative approaches or run tests in any other runner (like PowerShell or Command Line) with the TeamCity VSTest Adapter.

This article describes the specifics of the NUnit support in TeamCity and the alternative approaches to establishing it.

Supported NUnit Versions

The following NUnit versions are supported: 2.2.10, 2.4.1, 2.4.6, 2.4.7, 2.4.8, 2.5.0, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.5.9, 2.5.10, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 3.0.

It is possible to have several versions of NUnit installed on an agent machine and use any of them in a build.

NUnit Framework Compatibility

The following table represents the compatibility of TeamCity runners with the .NET implementations:

.NET Framework 1, 1.1, 2, 3.5

.NET Framework 4+

.NET Core 1+ and .NET 5+

NUnit runner

  • nunit-console.exe

  • nunit3-console.exe

check.png

check.png

error.png

.NET runner

  • dotnest test

  • dotnet msbuild /t:VSTest

  • dotnet vstest

error.png

check.png

check.png

VSTest adapter in other runners

error.png

check.png

check.png

Alternative Approaches

If using the NUnit or .NET build runners is inapplicable, TeamCity provides the following ways to configure NUnit tests reporting in TeamCity:

Feature Comparison for Alternative Approaches

Approach

Real-time Reporting

Execution in CLI-based runners

Tests Reordering

Implicit TeamCity .NET Coverage

<nunit2> NAnt task

check.png

check.png/error.png*

check.png

check.png

<NUnit> MSBuild task

check.png

check.png/error.png*

check.png

check.png

<NUnitTeamCity> MSBuild task

check.png

check.png/error.png*

check.png

check.png

TeamCity Add-in for NUnit

check.png

error.png

error.png

error.png

TeamCity NUnit Test Launcher

check.png

error.png

check.png

check.png

XML Reporting Plugin

error.png

check.png

N/A

N/A

* TeamCity-provided tasks may have different syntax/behavior. Some workarounds may be required to run the script without TeamCity.

In addition to the common test reporting features, TeamCity allows running NUnit tests under the x86 process on the x64 machine by introducing an explicit specification of the platform and runtime environment versions. You can define whether to use .NET Framework 1.1, 2.0 or 4.0 started under an MSIL, x64 or x86 platform.

NUnit Test Launcher

TeamCity provides its own NUnit tests launcher that can be used from the command line. The tests are run according to the passed parameters and, if the process is run inside the TeamCity build agent environment, the results are reported to the TeamCity agent.

You can pass the following command-line options to the TeamCity NUnit Test Launcher:

${teamcity.dotnet.nunitlauncher} <.NET Framework> <platform> <NUnit vers.> [/category-include:<list>] [/category-exclude:<list>] [/addin:<list>] <assemblies to test>

Option

Description

<.NET Framework>

Version of .NET Framework to run tests. Acceptable values are v1.1, v2.0, v4.0, and ANY.

<platform>

Platform to run tests. Acceptable values are x86, x64, and MSIL.

For .NET Framework 1.1, only MSIL option is available.

<NUnit vers.>

The test framework to use. The value has to be specified in the following format: NUnit-<version>.

/category-include:<list>

The list of categories separated by , (optional).

/category-exclude:<list>

The list of categories separated by , (optional).

/addin:<list>

List of third-party NUnit add-ins to use (optional).

<assemblies to test>

List of assembly paths separated by ; or whitespace.

/runAssemblies:processPerAssembly

Specify to run each assembly in a new process.

Category Expression

Beginning with NUnit 2.4.6 and up to but not including NUnit v3.0, a category expression can be used. This table shows some examples:

Expression

Action

A|B|C

Selects tests having any of the categories A, B or C.

A,B,C

Selects tests having any of the categories A, B or C.

A+B+C

Selects only tests having all three categories assigned.

A+B|C

Selects tests with both A and B OR with category C.

A+B-C

Selects tests with both A and B but not C.

-A

Selects tests not having category A assigned.

A+(B|C)

Selects tests having both category A and either of B or C.

A+B,C

Selects tests having both category A and either of B or C.

Note: As shown by the last two examples, the comma operator (,) is equivalent to the pipe (|) but has a higher precedence. The order of evaluation is as follows:

  1. Unary exclusion operator (-).

  2. High-precedence union operator (,).

  3. Intersection and set subtraction operators (+ and binary -).

  4. Low-precedence union operator (|).

Since the operator characters have special meaning, avoid creating a category that uses any of them in its name. For example, the category db-tests must not be used in the command line, as it appears to mean "run category db, except for category tests". The same limitation applies to the characters having a special meaning for the shell you are using.

The following examples assume that the teamcity.dotnet.nunitlauncher property is set as a system property on the Parameters page of Build Configuration Settings.

Run tests from an assembly:

%teamcity.dotnet.nunitlauncher% v2.0 x64 NUnit-2.2.10 Assembly.dll

Run tests from an assembly with the NUnit categories filter:

%teamcity.dotnet.nunitlauncher% v2.0 x64 NUnit-2.2.10 /category-include:C1 /category-exclude:C2 Assembly.dll

Run tests from assemblies:

%teamcity.dotnet.nunitlauncher% v2.0 x64 NUnit-2.5.0 /addin:Addin1.dll;Addin2.dll Assembly.dll Assebly2.dll

NUnit for NAnt Build Runner

This section assumes, that you already have a NAnt build script with the configured nunit2 task in it, and want TeamCity to track test reports without making any changes to the existing build script. Otherwise, consider adding the NUnit build runner as one of the steps for your build configuration.

To track tests defined in a NAnt build via the standard nunt2 task, TeamCity provides a custom task implementation and automatically replaces the original <nunit2> task with its own task. When the build is triggered, TeamCity starts TeamCity NUnit Test Launcher using own implementation of <nunit2>. This allows you to leave your the build script without changes and receive on-the-fly test reports in TeamCity.

If you don't want TeamCity to replace the original nunit2 task, consider the following options:

  • Use the NUnit console with TeamCity Add-in for NUnit.

  • Import XML test results via the XML Test Report plugin.

  • Use the command-line NUnit Test Launcher.

  • Configure reporting tests manually via service messages.

  • To disable nunit2 task replacement, set the teamcity.dotnet.nant.replaceTasks system property to false.

The nunt2 task implementation in TeamCity supports additional options that can be specified either as NAnt <property> tasks in the build script, or as system properties under Build Configuration | Build Parameters.

The following options are supported for the TeamCity <nunit2> task implementation:

Property

Description

teamcity.dotnet.nant.nunit2.failonfailureatend

Runs all tests regardless of the number of failed ones. Fails if at least one test has failed.

teamcity.dotnet.nant.nunit2.platform

Sets desired runtime execution mode for .NET 2.0 on x64 machines. Supported values are x86, x64, and ANY(default).

teamcity.dotnet.nant.nunit2.platformVersion

Sets the required .NET Framework version. Supported values are v1.1, v2.0, v4.0. The default value is equal to the NAnt target framework.

teamcity.dotnet.nant.nunit2.version

Specifies which version of the NUnit runner to use. The value has to be specified in the following format: NUnit-<version>.

It is possible to have several versions of NUnit installed on an agent machine and use any of them in a build.

teamcity.dotnet.nant.nunit2.addins

Specifies the list of third-party NUnit add-ins used for NAnt build runner.

teamcity.dotnet.nant.nunit2.runProcessPerAssembly

Set true if you want to run each assembly in a new process.

TeamCity NUnit test launcher will run tests in the .NET Framework, which is specified by the NAnt target framework: that is, on .NET Framework 1.1, 2.0 or 4.0 runtime. TeamCity also supports test categories for the <nunit2> task.

Adding the listed properties to the NAnt build script makes it TeamCity-dependent. To avoid this, specify properties as system properties under Build Configuration, or consider adding the <if> task.

Examples

Start tests form a single assembly files under x64 mode on .NET 2.0.

<property name="teamcity.dotnet.nant.nunit2.platform" value="x64" /> <nunit2> <formatter type="Plain" /> <test assemblyname="MyProject.Tests.dll" /> </nunit2>

Run all tests from category C1, but not C2.

<nunit2 verbose="true" haltonfailure="false" failonerror="true"> <formatter type="Plain" /> <test> <assemblies> <include name="dll.dll" /> </assemblies> <categories> <include name="C1" /> <exclude name="C2"/> </categories> </test> </nunit2>

Explicitly specify the version on NUnit to run tests with.
Note that in this case, the following property should be added before the nunit2 task call.

<property name="teamcity.dotnet.nant.nunit2.version" value="NUnit-2.4.10" /> <nunit2> <!--....--> </nunit2>

Using NUnit for MSBuild

This section describes how to use NUnit from MSBuild.

Working with NUnit Task in MSBuild Build

This section assumes that you already have an MSBuild build script with a configured NUnit task in it, and want TeamCity to track test reports without making any changes to the existing build script. Otherwise, consider adding the NUnit build runner as one of the steps of your build configuration.

Using NUnitTeamCity task in MSBuild Build Script

TeamCity provides a custom NUnitTeamCity task compatible with the NUnit task from MSBuild Community tasks project. If you provide the NUnitTeamCity task in your build script, TeamCity will launch its own test runner based on the options specified within the task. Thus, you do not need to have any NUnit runner, because TeamCity will run the tests.

To use the NUnitTeamCity task correctly:

  • Make sure the teamcity_dotnet_nunitlauncher system property is accessible on build agents. Build agents running Windows should automatically detect these properties as environment variables. If you need to set them manually, see defining agent-specific properties for more information.

  • Configure your MSBuild build script with the NUnitTeamCity task using the following syntax:

    <UsingTask TaskName="NUnitTeamCity" AssemblyFile="$(teamcity_dotnet_nunitlauncher_msbuild_task)" /> <NUnitTeamCity Assemblies="@(assemblies_to_test)" />

The NUnitTeamCity task supports the following attributes:

Property

Description

Platform

Execution mode on a x64 machine. Supported values are: x86, x64, and ANY.

RuntimeVersion

.NET Framework to use: v1.1, v2.0, v4.0, ANY. By default, the MSBuild runtime is used. Default is v2.0 for MSBuild 2.0 and 3.5. For MSBuild 4.0 the default value is v4.0.

IncludeCategory

As used in the NUnit task from the MSBuild Community tasks project.

ExcludeCategory

As used in the NUnit task from the MSBuild Community tasks project.

NUnitVersion

Version of NUnit to be used to run the tests.

To use NUnit 3.0 and above, see the section below.

Addins

List of third-party NUnit add-ins to be used. For more information on using NUnit add-ins, refer to the NUnit Add-ins Support section.

HaltIfTestFailed

True to fail task, if any test fails.

Assemblies

List of assemblies to run tests with.

RunProcessPerAssembly

Set true, if you want to run each assembly in a new process.

The custom TeamCity NUnit task also supports additional attributes. For the list of available attributes refer to this section.

If you need the TeamCity test runner to support third-party NUnit add-ins, refer to the NUnit Add-ins Support section for the details.

Example (part of the MSBuild build script):

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <UsingTask TaskName="NUnitTeamCity" AssemblyFile="$(teamcity_dotnet_nunitlauncher_msbuild_task)"/> <Target Name="SayHello"> <NUnitTeamCity Assemblies="!!!*put here item group of assemblies to run tests on*!!!"/> </Target> </Project>

Important notes

  • Be sure to replace . with _ when using system properties in MSBuild scripts. For example, use teamcity_dotnet_nunitlauncher_msbuild_task instead of teamcity.dotnet.nunitlauncher.msbuild.task.

  • TeamCity also provides Visual Studio Solution Runner for solution files of Microsoft Visual Studio 2005 and above. It allows you to use MSBuild-style wildcards for the assemblies to run unit tests on.

Examples

Run NUnit tests using specific NUnit runner version:

<Target Name="build_01"> <!-- start tests for NUnit-2.2.10 --> <NUnitTeamCity Assemblies="@(TestAssembly)" NUnitVersion="NUnit-2.2.10"/> <!-- start tests for NUnit-2.4.6 --> <NUnitTeamCity Assemblies="@(TestAssembly)" NUnitVersion="NUnit-2.4.8"/> </Target>

Run NUnit tests with custom add-ins with NUnit 2.4.6:

<Target Name="build"> <NUnitTeamCity Assemblies="@(TestAssembly)" Addins="NUnitExtension.RowTest.AddIn.dll" NUnitVersion="NUnit-2.4.6"/> </Target>

Run NUnit tests with custom add-ins with NUnit 2.4.6 in per-assembly mode:

<Target Name="build"> <NUnitTeamCity Assemblies="@(TestAssembly)" Addins="NUnitExtension.RowTest.AddIn.dll" NUnitVersion="NUnit-2.4.6" RunProcessPerAssembly="True"/> </Target>

To make a TeamCity independent build script, consider the following option:

<NUnitTeamCity ... Condition=" '$(TEAMCITY_VERSION)' != '' "/>

The MSBuild property TEAMCITY_VERSION is added to MSBuild when started from TeamCity.

Working with NUnit 3.0

Starting from version 3.0, NUnit supports TeamCity natively, so there is no need to use a special task for MSBuild as it was done for the earlier NUnit versions. The simplest way is to run the NUnit console via the standard Exec task.

The Getting Started with NUnit article contains details and examples.

MSBuild Service Tasks

For MSBuild, TeamCity provides the following service tasks that implement the same options as the Build Script Interaction:

TeamCitySetBuildNumber allows changing the build number:

<TeamCitySetBuildNumber BuildNumber="1.3_{build.number}" />

It is possible to use {build.number} as a placeholder for older build number.

TeamCityProgressMessage allows writing a progress message:

<TeamCityProgressMessage Text="Progress message text" />

TeamCityPublishArtifacts allows publishing all artifacts from the MSBuild item group:

<ItemGroup> <Files Include="*.dll" /> </ItemGroup> <TeamCityPublishArtifacts SourceFiles="@(Files-> '%(FullPath)' )" Condition=" '$(TEAMCITY_VERSION)' != '' "/>

TeamCityReportStatsValue is a handy task for publishing statistic values:

<TeamCityReportStatsValue Key="StatsValueType" Value="42" />

TeamCityBuildProblem reports a build problem which actually fails the build. Build problems appear on the Build Results page and also affect build status text:

<TeamCityBuildProblem description="description" identity="identity"/>
  • The mandatory description attribute is a human-readable text describing the build problem. By default, description appears in the build status text.

  • identity is an optional attribute and characterizes a particular build problem instance. It shouldn't change throughout builds if the same problem occurs: for example, the same compilation error. It should be a valid Java ID up to 60 characters long. By default, identity is calculated based on description.

TeamCitySetStatus is a task to change current build status text.

<TeamCitySetStatus Status="<status value>" Text="{build.status.text} and some aftertext" />

{build.status.text is substituted with an older status text. The status can have the SUCCESS value.

NUnit Add-ins Support

NUnit Add-in is an extension that plugs in to the NUnit core and changes the way it operates. Refer to the NUnit add-ins page for more information. This section covers description of the NUnit add-ins support for NAnt, MSBuild, and NUnit Console Launcher.

Using Add-ins in NAnt Build Runner

To support NUnit add-ins in the NAnt build runner, you need to add the teamcity.dotnet.nant.nunit2.addins property to your build script:

<property name="teamcity.dotnet.nant.nunit2.addins" value="<list of paths>" />

where <list> is the list of paths to NUnit add-ins separated by ;.

For example:

<property name="teamcity.dotnet.nant.nunit2.addins" value="../tools/addins/MyFirst.AddIn.dll;MySecond.AddIn.dll" />

Using Add-ins in TeamCity NUnit Console Launcher

To support NUnit add-ins for the console launcher, you need to provide the /addins:<list of addins separated with ;> command-line option.

For example:

${teamcity.dotnet.nunitlauncher} /addin:../tools/addins/MyFirst.AddIn.dll;nunit-addins/MySecond.AddIn.dll

Using Add-ins in MSBuild

This section is applicable to NUnit versions prior to 3.0.

To support NUnit add-ins for the MSBuild runner, specify the Addins property for the NUnitTeamCity task:

Addins="<list>"

where <list> is the list of add-ins separated by ; or ,.

For example:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="build"> <ItemGroup> <TestAssembly Include="$(MSBuildProjectDirectory)/MyTests.dll" /> </ItemGroup> <Target Name="build"> <NUnitTeamCity Assemblies="@(TestAssembly)" Addins="../tools/addins/MyFirst.AddIn.dll;nunit-addins/MySecond.AddIn.dll" /> </Target> </Project>

TeamCity Add-in for NUnit

TeamCity NUnit Add-in supports NUnit prior to version 3.0. For later versions, refer to the this section.

If you run NUnit tests via the NUnit console and want TeamCity to track the test results without having to launch the TeamCity test runner, the best solution is to use TeamCity Add-in for NUnit. You can plug this add-in into NUnit, and the tests will be automatically reported to the TeamCity server.

Alternatively, you can opt to use the XML Report Processing build feature, or manually configure test reporting by means of service messages.

To be able to review test results in TeamCity:

  1. In your build, set the path to the TeamCity Add-in to the system property teamcity.dotnet.nunitaddin (for MSBuild, it would be teamcity_dotnet_nunitaddin), and add the version of NUnit at the end of this path. For example:

    • For NUnit 2.4.X, use ${teamcity.dotnet.nunitaddin}-2.4.X.dll (for MSBuild: $(teamcity_dotnet_nunitaddin)-2.4.X.dll).
      Example for NUnit 2.4.7: NAnt: ${teamcity.dotnet.nunitaddin}-2.4.7.dll, MSBuild: $(teamcity_dotnet_nunitaddin)-2.4.7.dll.

    • For NUnit 2.5.0 alpha 4, use ${teamcity.dotnet.nunitaddin}-2.5.0.dll (for MSBuild: $(teamcity_dotnet_nunitaddin)-2.5.0.dll).

  2. Copy the .dll and .pdb TeamCity add-in files to the NUnit add-in directory.

Although you can copy these files once, it is highly recommended configuring your builds so that the TeamCity add-in files are copied to the NUnit add-in directory for each build, because these files could be updated by TeamCity.

The following example shows how to use the NUnit console runner with the TeamCity Add-in for NUnit 2.4.7 (on MSBuild):

<ItemGroup> <NUnitAddinFiles Include="$(teamcity_dotnet_nunitaddin)-2.4.7.*" /> </ItemGroup> <Target Name="RunTests"> <MakeDir Directories="$(NUnitHome)/bin/addins" /> <Copy SourceFiles="@(NUnitAddinFiles)" DestinationFolder="$(NUnitHome)/bin/addins" /> <Exec Command="$(NUnitHome)/bin/NUnit-Console.exe $(NUnitFileName)" /> </Target>

If you need to configure environment variables for NUnit explicitly, specify an environment variable with the value reference of %system.teamcity.dotnet.nunitaddin%. See this article for details.

NUnit 2.4.8 Issue
NUnit 2.4.8 has the following known issue: NUnit 2.4.8 runner tries to load an assembly according to the created AssemblyName object. However, the addins folder of NUnit 2.4.8 is not included in application probe paths. As a result, NUnit 2.4.8 fails to load any add-in in the console mode.
To solve the problem, we suggest that you use any of the following workarounds:

  • Copy the TeamCity add-in assembly both to the NUnit bin and bin/addins directories.

  • Patch NUnit-Console.exe.config to include the add-ins to application probe paths. Add the following code into the config/runtime element:

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">       <probing privatePath="addins"/> </assemblyBinding>
Last modified: 17 November 2022