Wednesday, December 02, 2009

NUnit for Visual Studio Addin

I recently stumbled upon this great addin for Visual Studio that uses the Visual Studio Test Adapter pattern to include NUnit tests within Visual Studio as MS Tests.  They appear in the Test List Editor and execute equivalent to MS Test, including those handy Run and Debug keyboard shortcuts I described in my last post.

Since they operate as MS Tests, the project requires some additional meta-data in the csproj file in order to have Visual Studio recognize this project as a Test library.  My last post has the details.

Curious to see how far the addin could act as a stand-in for NUnit, I fired up Visual Studio, a few beers, and the NUnit attribute documentation to put it through the works.  I’ve compiled my findings here in the table below.

In all fairness, there are a lot of attributes in NUnit, some of these you probably didn’t know existed.

NUnit Attribute Supported Comments
Category No Sadly, the addin does not register a new column definition for Category.  Though this feature is not tied to any functional behavior, it would be greatly welcomed to improve upon Visual Studio’s Test Lists.
Combinatorial Yes  
Culture / SetCulture No Tests that would normally be excluded by NUnit fail.
Datapoint / Theory No Test names do not match NUnit runtime.  All Datapoints produce result Not Runnable in the Test Results
Description No Value does not appear in the Test List Editor
Explicit No Explicit Tests are executed and appear as Enabled in the Test List Editor
ExpectedException Yes  
Ignore Partial Ignored tests are excluded from the Test List Editor, so they are ignored, but they do not appear as Enabled = False.
MaxTime / Timeout Partial Functions properly though supplied setting does not appear in the Timeout column in the Test List Editor
Platform No Tests are executed regardless of the specified platform.
Property - Custom properties do not appear in the output of the TRX file, which is where I’m assuming they would appear.  Not entirely sure if the schema would support custom properties however.
Random No Tests are generated, though the names contain only numbers.  Executing these tests produce the result Not Runnable in the Test Results.
Range No Tests are generated, though the names contain only numbers.  Executing these tests produce the result Not Runnable in the Test Results.
Repeat Yes  
RequiredAddin - Not tested.
RequiresMTA / RequiresSTA / RequiresThread Yes  
Sequential Yes  
Setup / Teardown Yes  
SetupFixture No Setup methods for a given namespace do not execute.
Suite - Not tested (requires command-line switch)
TestFixtureSetup / TestFixtureTeardown Yes  
Test Yes Of course!
TestCase Yes Tested different argument types (int, string), TestName, ExpectedException.
TestCaseSource Yes  

There’s quite a few No’s in this list, but the major players (Test, Setup/Teardown, TestFixtureSetup/Teardown) are functional.  I’m actually pleased that NUnit 2.5.2 features such as parameterized tests (TestCase, TestCaseSource) and Combinatorial / Sequential / Values are in place, as well as former addin features that were bundled into the framework (MaxTime / Repeat).

In respect to the malformed test names and non-runnable tests for the Theory / Range / Random attributes, hopefully this is a small issue that can be resolved.  The cosmetic issues with Ignore / Description / Category don’t pose any major concerns though they would be large wins in terms of full compatibility with MS Test user interface and features.

I’ve never used the SetupFixture nor the culture attributes, so I’m not losing much sleep over these.

However, for me, the main issue for me is that Explicit tests are always executed.  I’ve worked on many projects where a handful of tests either brought down the build server or couldn’t be run with other tests.  Rather than solve the problem, developers tagged the tests as Explicit – they work, but you better have a good reason to be running them.

Hats off to the NUnitForVS team.

submit to reddit


David said...

Nice find! Thanks for posting it. :)