Working with Build and Test Failures

During a build run, you can typically encounter one of the following failure types:

test failure — a failure of a unit or functional test that was attemted during a build.
build problem — any other issue that occurred during a build: inability to access a remote repository, failure to obtain or upload a build artifact, compilation error, and so on.

View Build Problems

In TeamCity, there are multiple ways to detect and investigate build problems.

Build Results Page

The Build Results Page displays the most detailed information about this individual build run. Its Overview tab displays expandable sections with a summary of both build and test problems occured during this run.

The first occurrence of a build problem adds "(new)" to the build's status message. This allows you to quickly distinguish unique and repeated problems when scrolling through a build history.

Click any problem to view a portion of a build log that corresponds to this issue.

Failed tests also show the Current failure / First failure toggle that allows you to quickly navigate to a build where the recurring issue first manifested.

If you are viewing the results of an older build while a newer build has completed without the same test failing, the failure will be crossed out. In this case, the Already Fixed tab that points to this newer build is visible.

If a test frequently changes its status from failed to successful and back, TeamCity marks such test as flaky. See the dedicated Flaky Tests section for more information.

Problems Tab

The project's Problems tab provides an overview of all currently active build problems, including those from child subprojects.

This page allows you to switch between build and test failures, as well as filter problems by their states.

Manage Build Problems and Test Failures

TeamCity's problem management capabilities revolve around two base concepts: investigations and mutes.

Investigations

Investigated problems are those that have TeamCity users assigned to them. These users are responsible for fixing corresponding build problems and/or test failures.

Users see a total number of investigations assigned to them in the TeamCity header. They can click this UI element to open the My Investigations page and view the detailed information about these investigations.

TeamCity draws a police officer icon to visualize investigated problems. Hovering overing this icon displays more information about the problem and allows users to reassign it or manually label it as fixed.

If a user marks an investigation as fixed, the icon changes its color to green. This icon will remain visible until a new runs finishes without encountering the same problem.

Manually setting the problem's status to "Fixed" reduces the user's investigation counter in the TeamCity header, but does not prevent newer builds from failing if the problem keeps emerging. If you want TeamCity to ignore this problem, mute it instead of labeling as fixed.

TeamCity can automatically create problem investigations and assign them to users who are most likely responsible for these failures. To do so, configure the Investigations Auto Assigner build feature.

Mutes

Muted problems are problems that do not prevent a build from reporting a successful finish. A build that failed due to a muted problem will appear as successfully finished, and display the number of muted problems.

You can mute build problems or tests to temporarily ignore a known issue that is expected to be resolved, to ignore tests that are expected to fail, or to ensure subsequent steps or Build Chain configurations run even if the build has encountered an issue.

TeamCity draws a loudspeaker icon to visualize muted problems. Hovering overing this icon displays more information about this issue.

Muting Failing Tests

Muting tests only affects builds that fail on the "At least one test failed" build failure condition but not on the other conditions. If any other failure condition applies to your build (for example, non-zero exit code), the build will still fail even if the failing tests are muted.

Your build script might need adjustment to make the build successful when there are failing but muted tests. Make sure the build does not fail because of other build failure conditions (for example, "Fail if build process exit code is not zero") in case the only errors encountered were tests failures. See the related issue TW-16784 for more details.

Manage Active Problems

Both Build Results Page and Problems Tab display UI elements that allow users to mute, unmute, investigate problems, mark them as fixed, or reassign them to different users. To apply the same action to multiple problems at once, go to the Problems tab and select required entries.

When editing investigations and mutes, TeamCity allows you to specify the following settings:

Investigation/mute scope. Choose "Project wide" to mute/investigate this problem in builds that belong to all configurations within the same project (including its child sub-projects). The "Selected build configuration" option allows you to mute/investigate a problem only for required configurations.
Project administrators can add the teamcity.internal.preferredInvestigationProject configuration parameter to a parent project to set this project as the default investigation/mute scope for all of its children. The parameter value must be set to the target project's ID. This property changes only the option suggested by default, users will still be able to choose a different scope.
Investigations and assignees. Muted problems make failed builds appear successful, which can obscure underlying issues. For that reason, we recommend pairing mutes with investigations and avoid keeping muted problems that are not investigated by anyone.
Automatic unmute/resolve policy. You can specify conditions under which TeamCity should unmute the problem or mark it as fixed.
- Automatically when fixed — if a problem is resolved in a newer build, TeamCity unmutes it and/or closes the related investigation. This is the default scenario that allows your team to quickly respond to the issue, should it re-appear in further builds.
  A "fixed" problem is either a problem whose status was manually changed (see the Investigations section), or a problem that went away in newer builds. See also: Relation with Branches.
- Manually — only TeamCity users can close active investigations and undo mutes. This mode is recommended for muting/investigating flaky tests, since these tests are unstable and a successful run cannot guarantee a problem is resolved.
- On a specific date — TeamCity closes the investigation or unmutes a problem on a given date. This condition is useful for non-critical issues that your team cannot fix right away, but you need the CI routines to keep running.
Copy settings to child subprojects. If you edit a investigation/mute that exists in this project and its child subprojects, TeamCity allows you to choose whether the updated settings should be propagated to these subprojects. Otherwise, investigations/mutes of child subprojects will differ (for example, assigned to a different user).

Required Permissions

To manage and view investigations/mutes in a project, a user must have the following permissions granted in this project:

"Mute/unmute problems in project" for managing mutes
"Assign/unassign investigation" for managing investigations
"View project and all parent projects" for viewing existing investigations/mutes

These permissions are granted to Project Administrators and System Administrators by default.

Relation with Branches

Investigations and mutes are not branch-specific. If a problem is investigated or muted, it is investigated/muted for all branches in which it manifests. However, TeamCity checks branch-specific builds to determine whether a problem is fixed. This affects how problems with "Automatically when fixed" unmute/close investigation conditions are handled.

If a failure occurs in multiple branches one of which is a default branch, the problem is considered fixed when it goes away in the default branch.
If a failure occurs in non-default active branches only, the problem is considered fixed when it goes away in all these branches.

Flaky Tests

Flaky tests are unstable tests that can finish both successful or failed with seemingly no differences between the build runs. TeamCity consideres a test flaky if one of the following is true:

A test has a high flip rate. A flip is one occurrence of a test changing its status from success to failure or back. TeamCity calculates the ratio of such flips to the total invocation count of a given test. This ratio is measured per agent, per build configuration, or over a certain time period (7 days by default). A test that constantly fails or succeeds has its flip rate close to zero. A test that "flips" every time it is invoked has the flip rate close to 100%.
A test flipped without underlying code changes. If a test finishes with an opposite status compared to the previous build run that processed the same revision (that is, no new changes were processed), TeamCity considers this test flaky.
A test invoked multiple times throughout one build finished with different statuses. This heuristic is supported for TestNG unit tests with invocationCount greater than 1, and tests carried out by the .NET runner with the Test retry count greater than 0.
There is a known issue with parameterized test invocations with TestNG: TeamCity relies on the Surefire and Failsafe Maven plugins for the purposes of test result reporting, and these plugins ignore test parameters in their XML output. As a result, parameterized test invocations within the same build are erroneously treated as multiple identical test invocations, and, if some of the invocations fail, the test is marked as flaky with the Flaky Failure reason. See the related issue.

A test is NOT considered flaky if:

multiple builds of different configurations are run on the same VCS change and the test results differ between these builds. Such results are most likely to be caused by environmental issues.
a VCS Root has its branches configured and a test flips in a non-default branch.

TeamCity displays flaky tests in the Tests tab of the Build Results Page...

...and on the Flaky tests page of a parent project.

For JUnit tests, a 3rd party tempus-fugit library can be used together with JUnit. It is sufficient to annotate a test with @Intermittent and use the IntermittentTestRunner test runner, as in the minimal example below:

import org.junit.Test;
import org.junit.runner.RunWith;

import com.google.code.tempusfugit.concurrency.IntermittentTestRunner;
import com.google.code.tempusfugit.concurrency.annotations.Intermittent;

@RunWith(IntermittentTestRunner.class)
public class MultipleViaTempusFugit {
  @Test
  @Intermittent(repetition = 10)
  public void test() {
    // ...
  }
}

If such a test flakes at least once at multiple invocations within a single build, the Flaky Failure heuristic will be triggered.

As with any failed test, you can assign investigations for a flaky test (or multiple tests). For flaky tests the resolution method is automatically set to "Manual"; otherwise the investigation will be automatically removed once the test is successful, which does not mean that the flaky test has been fixed.

Last modified: 12 July 2024