I recently stumbled upon this great addin for Visual Studio that uses the Visual Studio Test Adapter pattern to include NUnit tests within Visual Studio as MS Tests. They appear in the Test List Editor and execute equivalent to MS Test, including those handy Run and Debug keyboard shortcuts I described in my last post.
Since they operate as MS Tests, the project requires some additional meta-data in the csproj file in order to have Visual Studio recognize this project as a Test library. My last post has the details.
Curious to see how far the addin could act as a stand-in for NUnit, I fired up Visual Studio, a few beers, and the NUnit attribute documentation to put it through the works. I’ve compiled my findings here in the table below.
In all fairness, there are a lot of attributes in NUnit, some of these you probably didn’t know existed.
|Category||No||Sadly, the addin does not register a new column definition for Category. Though this feature is not tied to any functional behavior, it would be greatly welcomed to improve upon Visual Studio’s Test Lists.|
|Culture / SetCulture||No||Tests that would normally be excluded by NUnit fail.|
|Datapoint / Theory||No||Test names do not match NUnit runtime. All Datapoints produce result Not Runnable in the Test Results|
|Description||No||Value does not appear in the Test List Editor|
|Explicit||No||Explicit Tests are executed and appear as Enabled in the Test List Editor|
|Ignore||Partial||Ignored tests are excluded from the Test List Editor, so they are ignored, but they do not appear as Enabled = False.|
|MaxTime / Timeout||Partial||Functions properly though supplied setting does not appear in the Timeout column in the Test List Editor|
|Platform||No||Tests are executed regardless of the specified platform.|
|Property||-||Custom properties do not appear in the output of the TRX file, which is where I’m assuming they would appear. Not entirely sure if the schema would support custom properties however.|
|Random||No||Tests are generated, though the names contain only numbers. Executing these tests produce the result Not Runnable in the Test Results.|
|Range||No||Tests are generated, though the names contain only numbers. Executing these tests produce the result Not Runnable in the Test Results.|
|RequiresMTA / RequiresSTA / RequiresThread||Yes|
|Setup / Teardown||Yes|
|SetupFixture||No||Setup methods for a given namespace do not execute.|
|Suite||-||Not tested (requires command-line switch)|
|TestFixtureSetup / TestFixtureTeardown||Yes|
|TestCase||Yes||Tested different argument types (int, string), TestName, ExpectedException.|
There’s quite a few No’s in this list, but the major players (Test, Setup/Teardown, TestFixtureSetup/Teardown) are functional. I’m actually pleased that NUnit 2.5.2 features such as parameterized tests (TestCase, TestCaseSource) and Combinatorial / Sequential / Values are in place, as well as former addin features that were bundled into the framework (MaxTime / Repeat).
In respect to the malformed test names and non-runnable tests for the Theory / Range / Random attributes, hopefully this is a small issue that can be resolved. The cosmetic issues with Ignore / Description / Category don’t pose any major concerns though they would be large wins in terms of full compatibility with MS Test user interface and features.
I’ve never used the SetupFixture nor the culture attributes, so I’m not losing much sleep over these.
However, for me, the main issue for me is that Explicit tests are always executed. I’ve worked on many projects where a handful of tests either brought down the build server or couldn’t be run with other tests. Rather than solve the problem, developers tagged the tests as Explicit – they work, but you better have a good reason to be running them.
Hats off to the NUnitForVS team.