How to automatically fail TeamCity 4.0 build when MSTest 2008 webtests fail

Out of the box, there are two options to configure MSTest 2008 webtests with TeamCity 4.0 build server. The first option is to add webtest names to the MSTest assemblies list. The second option is to specify MSTest configuration and .vsmdi metadata files. In both cases, TeamCity 4.0 does not fail a project build when webtests fail. Nor does TeamCity display webtests results as it does for unit tests.

To solve this issue, we created a custom command-line file to execute MSTest webtests:

set ConfigPath=<Path to your webtest configuration and .vsmdi files> 

set ResultPath=<Path to .trx and .txt result files> 

set MSTestPath=<Path to your MSTest.exe> 

"%MSTestPath%\MSTest.exe" /runconfig:%ConfigPath%/Your.testrunconfig 

/testmetadata:%ConfigPath%/Your.vsmdi 

/resultsfile:%ResultPath%\Your%1.trx >> %ResultPath%\Your%1.txt 

if not %errorlevel% == 0 exit 1

This file takes a build number as the first positional parameter, executes webtests listed in Your.vsmdi, and saves .trx result file as well as the execution summary report so that TeamCity could store them as build artifacts. We then configured a dependent build configuration to run this command-line file as part of our build process.

Happy coding!

MSTest 2008 results file does not get refreshed when tests are executed in TeamCity 4.0

We run into a locking issue last week while setting up MSTest 2008 with TeamCity 4.0. We recommend using NUnit instead of MSTest, but MSTest 2008 is supported by TeamCity 4.0 and we did not anticipate any problems using it. The first build worked correctly as expected, but the second and all subsequent builds did not seem to execute any tests and did not overwrite MSTest results file created by the first build. All builds were displayed as successful and all included MSTest results from the first run!

I looked closer into the build logs and noticed the following entries:

"The results file 'C:\TeamCity\...\tests.trx' already exists. Please specify a different results file or verify the existing file is no longer needed and delete it."
"The command 'C:\...\mstest.exe' /testcontainer..." exited with code 1.

When I tried deleting the results file manually, I received the "Error Deleting File or Folder" exception. The file was locked by TeamCityAgentService-windows-x86-32.exe -> java.exe process.

To resolve this problem, we ended up naming our MSTest results file name as tests%env.BUILD_NUMBER%.trx. This way, each new build generates a brand-new results file. Note that this solution requires you to delete historical .trx files on a regular basis, which could be easily accomplished with a command line script.

Happy coding!