Running MS Team System tests in NUnit

Earlier this week I saw this question on Stack Overflow asking if it was possible to run MS Team System Tests under NUnit.  At first, this sounds like a really odd request.  After all, why not just convert your tests to NUnit and be done with it? 

I can think of a few examples where you may need interoperability – for example, two teams collaborating where team #1 has corporate policy for MSTest, while team #2 doesn’t have access to a version of Visual Studio with Team System.  Being able to target one platform but have the tests run in either environment is desirable.  Whatever the reason, it’s an interesting code challenge.

Fortunately, the good folks at Exact Magic Software published an NUnit addin that can adapt Team System test fixtures to NUnit.  Since NUnit identifies candidate fixtures simply by examining attribute names, the addin’s job is fairly straight forward, but it also recognizes MS specific features such as TestContext and DataSource driven tests.

Unfortunately, the NUnit adapter uses features that reside within the nunit.core assembly which ties it to a specific version of NUnit.  In this case, the addin will only work with NUnit 2.4.6.  This is a frustrating design problem that plagues many NUnit addins, including my own open source project.

Note: NUnit 3.0 plans to solve this problem by moving the boundary between framework and test runner so that each version of the framework knows how to run it’s own tests.  The GUI or host application will be able to load and execute the tests in a version independent manner.  Hopefully this means that addins will target and execute within a specific framework version, but will work in different versions of the user-interface.

The current version of NUnit (2.5.2) is a stepping stone between the 2.4.x framework and the upcoming overhauled 3.0 version.  As part of this transition, there’s a lot of breaking changes between older versions.  For me, this translates into a problem where Exact Magic’s project simply won’t compile if you update the dependencies.  Since I need to do similar work for my own project, this was good exercise.

I’ve reworked their code and put it here.

A few known caveats:

• The unit tests for the adapter attempt to perform reflection calls to internals fields or methods within the NUnit.Core that have been renamed or no longer exist.  I haven’t bothered fixing these tests.
• Does not appear to have support for AssemblyInitialize or AssemblyCleanup, though these could be adapted by adding an EventListener when the test run start and finish.  I may add this feature.
• The adapter doesn’t wrap functionality contained within MS Test, it simulates it’s behavior.  While it has the same result, it won’t capture the exact same nuances of Microsoft’s implementation.

If you have any concerns or questions, let me know.

Cheers.

My current approach to Context/Specification

One of my most popular posts from last year is an article on test naming guidelines, which was written to resemble the format used by the Framework Design Guidelines.  Despite the popularity of the article, I started to stray from those guidelines over the last year.  While I haven’t abandoned the philosophy of those guidelines, in fact most of the general advise still applies, I’ve begun to adopt a much different approach for structuring and organizing my tests, and as result, the naming has changed slightly too.

The syntax I’ve promoted and followed for years, let’s call it TDD or original-flavor, has a few known side-effects:

• Unnecessary or complex Setup – You declare common setup logic in the Setup of your tests, but not all tests require this initialization logic.  In some cases, a test requires entirely different setup logic, so the initial “arrange” portion of the test must undo some of the work done in the setup.
• Grouping related tests – When a complex component has a lot of tests that handle different scenarios, keeping dozens of tests organized can be difficult.  Naming conventions can help here, but are masking the underlying problem.

Over the last year, I’ve been experimenting with a Behavior-Driven-Development flavor of tests, often referred to as Context/Specification pattern.  While it addresses the side-effects outlined above, the goal of “true” BDD is to attempt to describe requirements in a common language for technical and non-technical members of an Agile project, often with a Given / When / Should syntax.  For example,

Given a bank account in good standing, when the customer requests cash, the bank account should be debited

The underlying concept is when tests are written using this syntax, they become executable specifications – and that’s really cool.  Unfortunately, I’ve always considered this syntax to be somewhat awkward, and how I’ve adopted this approach is a rather loose interpretation.  I’m also still experimenting, so your feedback is definitely welcome.

An Example

Rather than try to explain the concepts and the coding style in abstract terms, I think it’s best to let the code speak for itself first and then try reason my way out.

Note: I’ve borrowed and bended concepts from many different sources, some I can’t recall where.  This example borrows many concepts from Scott Bellware’s specunit-net.

public abstract class ContextSpecification
{
    [TestFixtureSetUp]
    public void SetupFixture()
    {
       BeforeAllSpecs();
    }
    
    [TestFixtureTearDown]
    public void TearDownFixture()
    {
       AfterAllSpecs();
    }

     [SetUp]
     public void Setup()
    {
       Context();
       Because();
    }

    [TearDown]
    public void TearDown()
    {
       CleanUp();
    }

    protected virtual void BeforeAllSpecs() { }
    protected virtual void Context() { }
    protected virtual void Because() { }
    protected virtual void CleanUp() { }
    protected virtual void AfterAllSpecs() { }   
}

public class ArgumentBuilderSpecs : ContextSpecification
{
    protected ArgumentBuilder builder;
    protected Dictionary<string,string> settings;
    protected string results;

    protected override void Context()
    {
       builder = new ArgumentBuilder();
       settings = new Dictionary<string,string>();
    }

    protected override void Because()
    { 
       results = builder.Parse(settings);
    }

    [TestFixture]
    public class WhenNoArgumentsAreSupplied : ArgumentBuilderSpecs
    {
       [Test]
       public void ResultsShouldBeEmpty()
       {
           results.ShouldBeEmpty();
       }
    }

    [TestFixture]
    public class WhenProxyServerSettingsAreSupplied : ArgumentBuilderSpecs
    {
       protected override void Because()
       {
           settings.Add("server", "proxyServer");
           settings.Add("port", "8080");

           base.Because();
       }

       [Test]
       public void ShouldContainProxyServerArgument()
       {
           results.ShouldContain("-DhttpProxy:proxyServer");
       }

       [Test]
       public void ShouldContainProxyPortArgument()
       {
           results.ShouldContain("-DhttpPort:8080");
       }
  }
}

Compared to my original flavor, this new bouquet has some considerable differences which may seem odd to an adjusted palate.  Let’s walk through those differences:

• No longer using Fixture-per-Class structure, where all tests for a class reside within a single class.
• Top level “specification” ArgumentBuilderSpecs is not decorated with a [TestFixture] attribute, nor does it contain any tests.
• ArgumentBuilderSpecs derives from a base class ContextSpecification which controls the setup/teardown logic and the semantic structure of the BDD syntax.
• ArgumentBuilderSpecs contains the variables that are common to all tests, but setup logic is kept to a minimum.
• ArgumentBuilderSpecs contains two nested classes that derive from ArgumentBuilderSpecs.  Each nested class is a test-fixture for a scenario or context.
• Each Test-Fixture focuses on a single action only and is responsible for its own setup.
• Each Test represents a single specification, often only as a single Assert.
• Asserts are made using Extension methods (not depicted in the code example)

Observations

Inheritance

I’ve never been a big fan of using inheritance, especially in test scenarios as it requires more effort on part of the future developer to understand the test structure.  In this example, inheritance plays a part in both the base class and nested classes, though you could argue the impact of inheritance is negated since the base class only provides structure, and the derived classes are clearly visible within the parent class.  It’s a bit unwieldy, but the payoff for using inheritance is found when viewing the tests in their hierarchy:

While technically you could achieve a similar effect by using namespaces to group tests, but you lose some of the benefits of encapsulation of test-helper methods and common variables.

Although we can extend our contexts by deriving from the parent class, this approach is limited to inheriting from the root specification container (ArgumentBuilderSpecs).  If you were to derive from WhenProxyServerSettingsAreSupplied for example, you would inherit the test cases from that class as well.  I have yet to find a scenario where I needed to do this.  While the concepts of DRY make sense, there’s a lot to be said about clear intent of test cases where duplication aids readability.

Extra Plumbing

There’s quite a bit of extra plumbing to be able to create our nested contexts, and it seems to take a bit longer to write tests.  This delay is either caused by grappling with new context/specification concepts, writing additional code for subclasses or more thought determining which contexts are required.  I’m anticipating that it gets easier with more practice, and some Visual Studio code snippets might simplify the authoring process.

One area where I can sense I’m slowing down is trying to determine if I should be overriding or extending Context versus Because.

Granular Tests

In this style of tests, where a class is created to represent a context, each context performs only one small piece of work and the tests serve as assertions against the outcome.  I found that the tests I wrote were concise and elegant, and the use of classes to structure the tests around the context helped to organize both the tests and the responsibility of the subject under test.  I also found that I wrote fewer tests with this approach – I write only a few principle contexts.  If a class starts to have too many contexts, it could mean that the class has too much responsibility and should be split into smaller parts.

Regarding the structure of the tests, if you’re used to fixture-per-class style of tests, it may take some time to get accustomed to the context performing only a single action.  Though as Steven Harman points out, visualizing the context setup as part of the fixture setup may help guide your transition to this style of testing.

Conclusion

I’m enjoying writing Context/Specification style tests as they read like a specification document and provide clear feedback of what the system does.  In addition, when a test fails, the context it belongs to provides additional meaning around the failure.

There appear to be many different formats for writing tests in this style, and my current format is a work in progress.  Let me know what you think.

Presently, I’m flip flopping between this format and my old habits.  I’ve caught myself a few times where I write a flat structure of tests without taking context into consideration – after a point, the number of tests becomes unmanageable, and it becomes difficult to identify if I should be tweaking existing tests or writing new ones.  If the tests were organized by context, the number of tests becomes irrelevant, and the focus is placed on the scenarios that are being tested.

Testing NUnit Addins from within NUnit

Note: This post refers to an internal feature of the NUnit.Core library and cannot be guaranteed in future versions.

Yesterday I released my first open source project, Selenium Toolkit for .NET, if you’re into functional web-testing I highly recommend you check it out.  I’d like to share a piece of code from that project – it’s either the biggest hack I’ve ever done or pretty clever.  Let me know what you think.

I don't like manually doing things more than once if I have to. Ironically, while building a tool to help automate functional web testing, I found that I had to do a lot of manual verification despite all the unit tests that I had written for my components. It started to get pretty frustrating: make a small change to my addin, uninstall and reinstall it, then open and run my sample test-fixtures to see how the NUnit framework had interpreted my changes. If I was going anywhere with this project, this was going to slow me down.

Fortunately, while spelunking through the NUnit.Core library, I found a really useful utility class that the fit the bill nicely and worked out pretty well. The TestAssemblyBuilder uses the nunit internals to find all the tests in an assembly and put them into a single Suite for inspection, execution, etc.

namespace Example
{
    using NUnit.Core.Builders;   

    public class ExampleTest
    {
        [Test]
        public void CanGetTestCount()
        {
            TestAssemblyBuilder assemblyBuilder = new TestAssemblyBuilder();
            TestSuite suite = assemblyBuilder.Build("AssemblyName.dll", false);

            Assert.AreEqual(4, suite.TestCount, "Not all tests were found.");
        }
    }
}

What’s really interesting here is the TestAssemblyBuilder uses a singleton within the NUnit internals, CoreExtensions.Host, to resolve all the core components for building up the suite.  This object implements the IExtensionHost interface which is also used to install our addin:

public interface NUnit.Core.Extensibility.IAddin
{
    bool Install(IExtensionHost host);
}

With this singleton at our disposal this is a pretty powerful option as I can now inject my addin into the framework with any configuration or mock that I want and test without environment dependencies.  In this example, I’m able to test how many times my Selenium instance would be started without ever launching a java host or browser:

namespace SeleniumToolkit.NUnit.Tests
{
    using NUnit.Core.Builders;
    using Selenium;
    using SeleniumToolkit;
    
    [TestFixture]
    public class FixtureBuilderTests
    {
        [Test]
        public void When_Fixture_Owns_Session_SeleniumFactory_Should_Only_Create_Sessions_For_Fixtures()
        {
            // create config that influences how the addin will work inside NUnit
            SeleniumConfiguration config = new SeleniumConfiguration();
            config.AddinSettings.RecyclePerFixture = true;
            
            // create basic event listener and suite-builder for installation into addin
            var eventListener = new SeleniumEventListener(null); // null = no java host
            var suiteBuilder = new SeleniumTestFixtureBuilder(config);

            // mock out the factory that creates our selenium objects
            var factory = MockRepository.GenerateMock<ISeleniumFactoryProvider>();
            var selenium = MockRepository.GenerateMock<ISelenium>();

            // tell our mock'd factory to always return our mock selenium
            factory.Expect(f => f.Create(null,0,null,null)).IgnoreArguments()
                   .Return(selenium).Repeat.Any();

            // set our expectations for how many times selenium will start/stop
            selenium.Expect(s => s.Start()).Repeat.Times(3);
            selenium.Expect(s => s.Stop()).Repeat.Times(3);

            // inject our mock into the toolkit
            SeleniumFactory.Configure(config, factory);

            // install addin
            SeleniumNUnitAddin addin = new SeleniumNUnitAddin();
            addin.Install(CoreExtensions.Host, suiteBuilder, eventListener); // injection method

            // using installed addin, test construction of fixtures
            var assemblyBuilder = new TestAssemblyBuilder();
            TestSuite suite = assemblyBuilder.Build( 
                                               "NUnit.TestAssembly.FixtureScenario.dll", 
                                               false);                        

            // run our sample fixtures against our configured runtime
            suite.Run(eventListener);

            // verify our behavior is as expected
            selenium.VerifyAllExpectations();
        }
    }

}

I should point out here that I don’t have my addin installed in NUnit’s addin folder. In fact, because I’m manually injecting it into NUnit, installation isn’t necessary. I also discovered that having the addin installed introduced some unexpected behavior with the tests in a few places.

To ensure that I'm testing my addin in isolation, it's important to clean up the CoreExtensions singleton after each test. While the NUnit core provides a mechanism to remove items from its extension points, it requires that you pass in the instances you want to remove, which didn’t quite fit my needs.  I suppose I could have implemented equality comparisons in my extensions, but I opted for the hack ‘em out using reflection option. It’s gross, but it works:

public class AddinUtil
{
    public static void Uninstall()
    {
        IExtensionPoint eventListeners = CoreExtensions.Host.GetExtensionPoint("EventListeners");
        IExtensionPoint suiteBuilders = CoreExtensions.Host.GetExtensionPoint("SuiteBuilders");
        IExtensionPoint testBuilders = CoreExtensions.Host.GetExtensionPoint("TestCaseBuilders");

        RemoveExtension(typeof(SeleniumEventListener), eventListeners);
        RemoveExtension(typeof(SeleniumTestFixtureBuilder), suiteBuilders);
        RemoveExtension(typeof(SeleniumTestCaseBuilder), testBuilders);
    }

    private static void RemoveExtension(Type type, IExtensionPoint extensionPoint)
    {
        ArrayList items = GetInternalExtensions(extensionPoint);
        ArrayList toRemove = new ArrayList();

        foreach (object item in items)
        {
            if (type.IsAssignableFrom(item.GetType()))
            {
                    toRemove.Add(item);
            }
        }

        foreach (object item in toRemove)
        {
            items.Remove(item);
        }
    }

    private static ArrayList GetInternalExtensions(IExtensionPoint extensionPoint)
    {
        object extensions = extensionPoint.GetType()
					  .GetField("extensions", 
							BindingFlags.NonPublic | 
							BindingFlags.Public | 
							BindingFlags.Instance)
					  .GetValue(extensionPoint);
        return (ArrayList)extensions;
    }
}

Conclusion

This technique provides a brief glimpse of how to use automate testing nunit addins inside nunit using dependency injection.  I’m fairly certain that I will still have to manually verify the addin’s behaviour from time to time, but with this approach I hopefully won’t have to do it that often.

Catching server errors with WatiN: redux

Stumbled upon this post about how to catch server errors for your WatiN tests.  The approach outlined provides a decent mechanism for detecting server errors by sub-classing the WatiN IE object.  While I do appreciate the ability to subclass, it bothers me a bit that I have to write the logic in my subclass to detect server errors.  After poking around a bit, I think there's a more generic approach that can be achieved by tapping into the NavigateError event of the native browser:

public class MyIE : IE
{
    private InternetExplorerClass ieInstance;
    private NavigateError error;

    public MyIE()
    {
        ieInstance = (InternetExplorerClass) InternetExplorer;
        ieInstance.BeforeNavigate += BeforeNavigate;
        ieInstance.NavigateError += NavigateError;
    }

    public override void WaitForComplete()
    {
        base.WaitForComplete();
        if (error != null)
        {
            throw new ServerErrorException(Text);
        }
    }

    void BeforeNavigate(string URL, int Flags, string TargetFrameName, ref object PostData, string Headers, ref bool Cancel)
    {
        error = null;
    }

    void NavigateError(object pDisp, ref object URL, ref object Frame, ref object StatusCode, ref bool Cancel)
    {
        error = new NavigateError(URL,StatusCode);
    }

    private class NavigateError
    {
        public NavigateError(object url, object statusCode)
        {
            _url = url;
            _statusCode = statusCode;
        }

        private object _url;
        private object _statusCode;
    }
}
public class ServerErrorException : Exception 
{
    public ServerErrorException(string message) : base(String.Format("A server error occurred: {0}",message))
    { } 
}

Few caveats:

• Constructor of MyIE needs to be updated to reflect the other constructor overloads.
• Need to ensure that URL of NavigateError is the same URL of BeforeNavigate
• Test library needs to reference the Interop.SHDocVw wrapper for Internet Explorer
• Only tested with IE7

While I wouldn't consider COM Interop to be a "clean" solution, it is more bit more portable between solutions.  And if it was this easy, why isn't it part of WatiN anyway?

Legacy Projects: Test the User Interface with Selenium or WatiN

Following up on the series of posts on Legacy Projects, my legacy project with no tests now has a build server with empty coverage data.  At this point, it's really quite tempting to start refactoring my code, adding in tests as I go, but that approach is slightly ahead of the cart.

Although Tests for the backend code would help, they can't necessarily guarantee that everything will work correctly.  To be fair, the only real guarantee for the backend code would be to write Tests for the existing code and then begin to refactor both Tests and code.  This turns out to be a very time consuming endeavour as you'll end up writing the Tests twice.  In addition, I'm working with the assumption that my code is filled with static methods with tight-coupling which doesn't lend itself well to testing.  I'm going to need a crowbar to fix that, and that'll come later.

It helps to approach the problem by looking at the current manual process as a form of unit testing.  It's worked well up to this point, but because it's done by hand it's a very time consuming process that is prone to error and subjective of the user performing the tests.  The biggest downfall of the current process is that when the going get's tough, we are more likely to miss details.  In his book, Test Driven Development by Example, Kent Beck refers to manual testing as "test as a verb", where we test by evaluating aspects of the system.  What we need to do is turn this into "test as a noun" where the test is a "procedure to evaluate" in an automated fashion.  By automating the process, we eliminate most of the human related problems and save a bundle of time. 

For legacy projects, the best place automation starting point is to test the user interface, which isn't the norm for TDD projects.  In a typical TDD project, user interface testing tends to appear casually late in the project (if it appears at all), often because the site is incomplete and the user interface is a very volatile place;  UI tests are often seen as too brittle.  However, for a legacy project the opposite is true: the site is already up and running and the user interface is relatively stable; it's more likely that any change we make to the backend systems will break the user interface.

There is some debate on the topic of where this testing should take place.  Some organizations, especially those where the Quality Assurance team is separated from the development teams, rely on automated testing suites such as Empirix (recently acquired by Oracle) to perform functional and performance tests.  These are powerful (and expensive) tools, but in my opinion are too late in the development cycle --  you want to catch minor bugs before they are released to QA, otherwise you'll incur an additional bug-fix development cycle.  Ideally, you should integrate UI testing into your build cycle using tools that your development team is familiar with.  And if you can incorporate your QA team into the development cycle to help write the tests, you're more likely to have a successful automated UI testing practice.

Of the user interface testing frameworks that integrate nicely with our build scripts, two favourites come to mind:  Selenium and WaitN.

Using Selenium

Selenium is a java-based powerhouse whose key strengths are platform and browser diversity, and it's extremely scalable.  Like most java-based solutions, it's a hodge-podge of individual components that you cobble together to suit your needs; it may seem really complex, but it's a really smart design.  At its core, Selenium Core is a set of JavaScript files that manipulate the DOM.  The most common element is known as Selenium Remote-Control, which is a server-component that can act as a message-broker/proxy-server/browser-hook that can magically insert the Selenium JavaScript into any site --  it's an insanely-wicked-evil-genius solution to overcoming cross-domain scripting issues.  Because Selenium RC is written in Java, it can live on any machine, which allows you to target Linux, Mac and PC browsers.  The scalability feature is accomplished using Selenium Grid, which is a server-component that can proxy requests to multiple Selenium RC machines -- you simply change your tests to target the URL of the grid server.  Selenium's only Achilles' heel is that SSL support requires some additional effort.

A Selenium test that targets the Selenium RC looks something like this:

[Test]
public void CanPerformSeleniumSearch()
{
    ISelenium browser = new DefaultSelenium("localhost",4444, "*iexplore", "http://www.google.com");
    browser.Start();
    browser.Open("/"); 
    browser.Type("q", "Selenium RC"); 
    browser.Click("btnG");

    string body = browser.GetBodyText();

    Assert.IsTrue(body.Contains("Selenium"));

    browser.Stop(); 
}

The above code instantiates a new session against the Selenium RC service running on port 4444.  You'll have to launch the service from a command prompt, or configure it to run as a service.  There are lots of options.  The best way to get up to speed is to simply follow their tutorial...

Selenium has a FireFox extension, Selenium IDE, that can be used to record browser actions into Selenese.

Using WatiN

WatiN is a .NET port of the java equivalent WatiR.  Although it's currently limited to Internet Explorer on Windows (version 2.0 will target FireFox), it has an easy entry-path and a simple API.

The following WatiN sample is a rehash of the Selenium example.  Confession: both samples are directly from the provided documentation...

[Test]
public void CanPerformWatiNSearch()
{
    using (IE ie = new IE("http://www.google.com"))
    {
        ie.TextField(Find.ByName("q")).TypeText("WatiN");
        ie.Button(Find.ByName("btnG")).Click();

        Assert.IsTrue(ie.ContainsText("WaitN");
    }
}

As WatiN is a browser hook, its API contains exposes the option to tap directly into the browser through Interop.  You may find it considerably more responsive than Selenium because the requests are marshaled via windows calls instead of HTTP commands.  Though there is a caveat to performance: WatiN expects a Single Threaded Apartment model in order to operate, so you may have to adjust your runtime configuration.

WatiN also has a standalone application, WatiN Recorder, that can capture browser activity in C# code.

UI Testing Strategy Tips

Rather than writing an exhaustive set of regression tests, here's my approach:

• Start Small: Begin by writing coarse UI tests that demonstrate simple functionality.  For example, a test that hits the homepage and validates that there aren't any 500 errors.  Writing complex tests that validate specific HTML markup take longer to produce and often tend to be brittle and less maintainable in the long run.
• Map out and test functional areas:  Identify the key functional elements of the site that QA would normally regression test for a build: login, update a profile, add items to a shopping cart, checkout, search, etc.  Some of these will be definite road-blockers that you'll have to work around -- you'll quickly realize you can't guarantee profile-ids and passwords between environments, or maybe your product catalog changes too frequently.  Some will require creative thinking, others may inspire custom testing tools that can perform test-specific queries or functions.  You may even find a missing need in the backend systems that you could build and leverage as part of your tests. 
• Write tests for functional changes:  You don't need to sit down an write an exhaustive site wide regression fixture -- focus on the areas that you touch.  If you write tests before you make any changes you can use these tests to help automate the debugging process.  The development effort is relatively small -- you have to test it anyway a dozen times by hand.
• Write tests for testing bugs!!!:  What better motivation could you have?  This is what regression testing is all about!
• Design for different environments:  The code examples above have URLs hard-coded.  Consider using a tool that uses configuration settings to retrieve or help construct URLs so that you can run your UI tests against your local instance, dev, build-server, QA, integration, etc.  UI Tests make great build-validation utilities!

Working with Legacy .NET Projects

My current project at work is a legacy application, written using .NET 1.1. The application is at least 5 years old and has had a wide range of developers. It's complex, has many third-party elements and constraints and lots of lots of code. Like all legacy applications, they set out with best of intentions but ended up somewhere else when new requirements started to deviate from the original design. It's safe to say that it's got challenges, it works despite its bugs and all hope is not yet lost.

Oh, and no unit Tests. Which in my world, is a pretty big thing. Hope you like Spaghetti!!

Fortunately, the client has agreed to a .NET 2.0 migration, which is a great starting place. All in all, I see this as a great refactoring exercise to slowly introduce tests and proper design. Along the way, we'll be fixing bugs, improving performance and reducing friction to change. I'll be writing some posts over the next while that talk about the strategies were using to change our landscape. Maybe, you'll find them useful for your project.

Related Posts:

• Get Statistics from your Build Server
• Coverage Data without Tests
• Test the User Interface
• Planning for Refactoring 

TDD Tips: Create Custom NUnit Categories

In my recent post about test naming conventions and guidelines, I mentioned that you should annotate tests for third-party and external dependencies with category attributes and limit the number of categories that you create. This post will show basic usage of categories, will explain some of the reasoning behind limiting the number of categories. I'll also show how you can create your own categories with NUnit 2.4.x.

Although it's possible to annotate all of your tests with categories, they're really only useful for marking sensitive tests, typically around logical boundaries in your application. Some of the typical categories that I mark my tests with:

• Database: Tests that require a database to execute. You might want to exclude these tests when you're working remotely or isolate these tests if you need to validate a database deployment for an environment.
• Integration: Tests that interact with external components you don't have much control over, such as web-services or other infrastructure.
• Web: Tests that perform regression tests on the visual aspect of a web-site. These tests tend to be very time consuming or require special configuration, so being able to exclude them until they're required can be a big help. Often I run these type of tests when the build server kicks off a build.

Usage

Using categories are very straight forward. Here's an example of a test that is marked with a "Database" category

namespace Code
{
  [TestFixture]
  public class AdoOrderManagementProvider
  {
      [Test,Category("Database")]
      public void CanRetrieveOrderById()
      {
          // database code here
      }
  }
}

Challenges with Categories

One problem I've found with using categories is that category names can be difficult to keep consistent in large teams, mainly because the category name is a literal value that is passed to the attribute constructor. In large teams, you either end up with several categories with different spellings, or the unclear intent of the categories becomes an obstacle which prevents developer adoption.

Fortunately, since NUnit 2.4.x, it's possible to create your own custom categories by deriving from the CategoryAttribute class. (In previous releases, the CategoryAttribute class was sealed.) Creating your own custom categories as concrete classes allows the solution architect to clearly express the intent of the testing strategy, and relieves the developer of spelling mistakes. As an added bonus, you get Intellisense support (through Xml Documentation syntax), ability to identify usages and the ability to refactor the category much more effectively than a literal value.

Here's the code for a custom database category, and the above example modified to take advantage of it:

using NUnit.Framework;

namespace NUnit.Framework
{
  /// <summary>
  /// This test, fixture or assembly has a direct dependency to a database.
  /// </summary>
  [AttributeUsage(AttributeTarget.Class | AttributeTarget.Method | AttributeTarget.Assembly, AllowMultiple = false)]
  public class RequiresDatabaseAttribute : CategoryAttribute
  {
      public RequiresDatabaseCategoryAttribute() : base("Database")
      {}
  }
}

namespace Code
{
  [TestFixture]
  public class AdoOrderManagementProvider
  {
      [Test, RequiresDatabase]
      public void CanRetrieveOrderById()
      {
          // etc...
      }
  }
}

It's important to point out that categories can be applied per Test, per Fixture or even for the entire Assembly, so you have lots of options in terms of the level of granularity.

Filtering Tests using Categories

The real advantage to using categories is that you can filter which tests should be included or excluded when the tests are run.

Filtering Categories within Nunit-Gui.exe

To actively include/exclude tests by category in the GUI:

1. Click on the Categories tab in the top left
2. Select the categories you wish to include/exclude, then click the Add button.
3. If excluding tests, check the "exclude these categories" checkbox.

Filtering Categories in NUnit 2.4.7.

Filtering Categories with Nunit-Console.exe

To include/exclude tests by category from the command line use either the /include:<category-name> or /exclude:<category-name> parameters. It's possible to provide a list of categories by using a comma delimiter.

Example of running all tests within assemblyName.dll except for tests marked as Database or Web.:

nunit-console assemblyName.dll /exclude:Database,Web

Example of running only tests marked with the Database category:

nunit-console assemblyName.dll /include:Database

Note: The name of the category is case-sensitive.

Code Available

I'm pleased to announce that I've setup a repository using Google Project hosting. I'll be posting downloadable code samples. I've created a few simple NUnit categories based on the examples above that you can download and use for your projects:

• Compiled zip
• Source

Happy testing!

TDD Tips: Test Naming Conventions & Guidelines

The idea behind test driven development is that you are writing the test first. Since all code must reside in a method, the very first step before you can write any code, is to name the test. If you're new to TDD, you'll find this to be a very difficult thing to do. Don't let this discourage you, I'd go so far to say that out of all the tasks a developer must accomplish, finding names for things is perhaps the most difficult. W.H. Auden's statement show's that this "meta" thing transcends development:

Proper names are poetry in the raw. Like all poetry they are untranslatable. ~W.H. Auden

This begs a question that comes up frequently for new TDD developers starting out as well as experienced developers during code review: "Is there a naming convention or guidelines for unit tests?" Some believe it to be a black art, but I think it's more like acquiring a rhythm and following along. Once you've got the rhythm it gets easier.

Prior to diving into the guidelines, let's clear up some basic vocabulary:

• Target / Subject: I often use the term "Target" or "Subject" to refer to the piece of functionality that I'm testing.
• Fixture: Synonymous with "TestFixture", a fixture is a class that contains a set of related tests. Fixtures are classes that have been decorated with the [TestFixture] attribute.
• Suite: Test Suites are an older style of organizing tests. They're specialized fixtures that programmatically define which Fixtures or Tests to run. NUnit supports them for backward compatibility by using the [TestSuite] attribute. Since NUnit dynamically finds all tests with the [TestFixture] attribute, they're not as popular these days.
• Test: You may have noticed that I capitalize Test in all my entries. Tests are methods within the Fixture that are decorated with the [Test] attribute and contain code that validates the functionality of our target.
• Setup/TearDown: Test Fixtures can designate a special piece of code to run before every Test within that Fixture. That method is decorated with the [Setup] attribute. Likewise, a method with the [TearDown] attribute is called at the end of every test within a fixture.
• Fixture Setup/Fixture TearDown: Similar to constructors and finalizers, methods with the [TestFixtureSetup] or [TestFixtureTearDown] attributes execute before and after the Fixture executes. These methods happen before [Setup] and after [TearDown].
• Category : The [Category] attribute when applied to a method associates the Test within a user-defined category.
• Ignore: Tests with the [Ignore] attribute are skipped over when the Tests are run.
• Explicit: Tests with the [Explicit] attribute won't run unless you manually run them.

The following are some suggestions I've adopted or recommended to others from past projects. Feel free to take 'em at face value, or leave a comment if you have some to add:

Fixtures

DO: Name Fixtures consistently
TestFixtures should follow a consistent naming convention to make tests easier to find. Choose a naming convention such as <TargetType>Fixture or <TargetType>Test and stick to it.

DO: Mimic namespaces of Target Code
To help keep your tests organized, use the same folders and namespace structures as your target assembly. This will help you locate tests for target types and vice versa. Since most Test runners group Tests by their namespace, it's really easy to run all tests for a specific namespace by selecting by the container folder -- which is great for regression testing an area of code. I've got another post which talks about how to structure your Test namespaces.

DO: Name Setup/TearDown methods consistenty
When naming your fixture setup and teardown methods, you really should pick a style for these methods and stick with it. Personally, I can't find any reason why you would deviate from naming these methods FixtureSetup, FixtureTearDown, Setup, and TearDown as these provide clear names. By following a standard TestFixtures structure you can cut down some of the visual noise, make tests easier to read and produce more maintainable tests across multiple developers.

CONSIDER: Separating your Tests from your Production Code
As a general rule, you should try to separate your tests from your production code. If you have a requirement where you want to test in production or verify at the client's side, you can accomplish this simply by bundling the test library with your release. Still, every project is different, and tests won't necessarily impede production other than bloating up your assembly. Separate when needed, and use your gut to tell you when you should.

CONSIDER: Deriving common Fixtures from a base Fixture
In scenarios where you are testing sets of common classes or when tests share a great deal of duplication, consider creating a base TestFixture that your Fixtures can inherit.

CONSIDER: Using Categories instead of Suites or Specialized Tests
Although Suites can be used to organize Tests of similar functionality together, Categories are the preferred method. Suites represent significant developer overhead and maintenance. Likewise, creating specialized folders to contain tests (ie "Database Tests") also creates additional effort as tests for a particular Type become spread over the test library. Categories offer a unique advantage in the UI and at the command-line that allows you to specify which categories should be included or excluded from execution. For example, you could execute only "Stateful" tests against an environment to validate a database deployment.

CONSIDER: Splitting Test Libraries into Multiple Assemblies
From past experience, projects go to lengths to separate tests from code but don't place a lot of emphasis on how to structure Test assemblies. Often, a single Test library is created, which is suitable for most projects. However, for large scale projects that can have hundreds of tests this approach can get difficult to manage. I'm not suggesting that you should religiously enforce test structure, but there may be logical motivators to divide your test assemblies into smaller units, such as grouping tests with third-party dependencies or as an alternative for using Categories. Again, separate when needed, and use your gut to tell you when you should. (You can always go back)

AVOID: Empty Setup methods
As a best practice, you should only write the methods that you need today. Adding methods for future purposes only adds visual noise for maintenance purposes. The exception to this is when you are creating a base Fixture that contains empty methods that will be overridden by derived classes.

Tests

DO: Name Tests after Functionality
The test name should match a specific unit of functionality for the target type being tested. Some key questions you may want to ask yourself: "what is the responsibility of this class?" "What does this class need to do?" Think in terms of action words. Well written test names should provide guidance when the test fails. For example, a test with the name CanDetermineAuthenticatedState provides more direction about how authentication states are examined than Login.

DO: Document your Tests
You can't assume that all of your tests will be intuitive for everyone who reviews them. Most tests require special knowledge about the functionality your testing, so a little documentation to explain what the test is doing is helpful. Using XML Documentation syntax might be overkill, but a few comments here and there are often just the right amount to help the next person understand what you need to test and how your test approaches demonstrates that functionality.

CONSIDER: Use "Cannot" Prefix for Expected Exceptions
Since Exceptions are typically thrown when your application is a performing something it wasn't designed to do, prefix "Cannot" to tests that are decorated with the [ExpectedException] attribute. Some examples: CannotAcceptNullArguments, CannotRetrieveInvalidRecord.

I would consider this a "DO" recommendation, but this a personal preference. I can't think of scenarios where this isn't the case, so this one is up for debate.

CONSIDER: Using prefixes for Different Scenarios
If your application has features that differ slightly for application roles, it's likely that your test names will overlap. Some teams have adopted a For<Scenario> syntax (CanGetPreferencesForAnonymousUser). Other teams have adopted an underscore prefix _<Scenario> (AnonymousUser_CanGetPreferences).

AVOID: Ignore Attributes with no explanation
Tests that are marked with the Ignore attribute should include a reason for why this test has been disabled. Eventually, you'll want to circle back on these tests and either fix them or alter them so that they can be used. But without an explaination, the next person will have to do a lot of investigative work to figure out that reason. In my experience, most tests with the Ignore attribute are never fixed.

AVOID: Naming Tests after Implementation
If you find that your tests are named after the methods within your classes, that's a code smell that you're testing your implementation instead of your functionality. If you changed your method name, would the test name still make sense?

AVOID: Using underscores as word-separators
I've seen tests that use_underscores_as_word_separators_for_readability, which is so-o-o 1960. PascalCase should suffice. Imagine all the time you save not holding down the shift key.

AVOID: Unclear Test Names
Sometimes we create tests for bugs that are caught late in the development cycle, or tests to demonstrate requirements based on lengthy requirements documentation. As these are usually pretty important tests (especially for bugs that creep back in), it's important to avoid giving them vague test names that represent a some external requirement like FixForBug133 or TestCase21.

Categories

DO: Limit the number of Categories
Using Categories is a powerful way to dynamically separate your tests at runtime, however their effectiveness is diminished when developers are unsure which Category to use.

CONSIDER: Defining Custom Category Attributes
As Categories are sensitive to case and spelling, you might want to consider creating your own Category attributes by deriving from CategoryAttribute. UPDATE: Read more about custom NUnit Categories.

Well, that's all for now. Are you doing things differently, or did I miss something? Feel free to leave a comment.

Updates:

• 6/18/08 - Added links to custom NUnit Categories

TDD Tips: Getting value out of Code Coverage

If you're following true test driven development, you should be writing tests before you write the code. By definition you only write the code that is required and you should always have 100% code coverage.

Unfortunately, this is not always the case. We have legacy projects without tests; we're forced to cut corners; we leave things to finish later that we forget about. For that reason, we look to tools to give us a sense of confidence in the quality of our code. Code coverage is often (dangerously) seen as a confidence gauge. So to follow up on a few of my other TDD posts, I want to talk about what value code coverage can provide and how you should and shouldn't use it...

Let's start by looking at what code coverage will tell us...

• Code coverage shows which parts of our code have been tested. This metric is usually inferred as a total percentage of code that has been tested.
• Most coverage tools keep track of how many times methods have been visited. This value shows us how much or how little testing is represented for specific a code block, but as far as I know, there's no overall valuable metric. You could infer "top most tested" or "top least tested" metrics.

In some cases, code coverage can be used to contribute to a confidence level. I feel better about a large code base that has an 80% coverage than little or no coverage. But coverage is just statistical data -- it can be misleading...

Good Coverage doesn't mean Good Code
Having a high coverage metric cannot be used as an overall code quality metric. Code coverage cannot reveal that your code or tests haven't accounted for unexpected scenarios, so it's possible that buggy code with "just enough" tests can have high coverage.

Good Coverage doesn't mean Good Tests
A widely held belief of TDD is that the confidence level of the code is proportional to the quality of the tests. Code coverage tools can be very useful to developers to identify areas of the code that are missing tests, but should not be used as a benchmark for test quality. Tests can become meaningless when developers write tests to satisfy coverage reports instead of writing tests to prove functionality of the application. See the example below.

How a few bad tests ruin coverage

Developers can unknowingly write a test that invalidates coverage. To demonstrate, let's assume we have a really simple Person class. For sake of argument, FirstName is always required so we make it available through the the constructor.

[TestFixture]
public class PersonTest
{
   [Test]
   public void CanCreatePerson()
   {
      Person p = new Person("Bryan");
      Assert.AreEqual(p.FirstName,"Bryan");
   }
}

public class Person
{
    public Person(string firstName)
    {
      _first = firstName;
    }

    public virtual string FirstName
    {
      get { return _first; }
      set { _first = value; }
    }

    private string _first;
}

This is all well and good. However, a code coverage report would reveal that the FirstName property setter (highlighted above) has no coverage.

Should we fix the code....

public Person(string firstName)
{
   _first = firstName;
   FirstName = firstName; // virtual method call in constructor 
                          // is a FxCop violation
}

... or the test?

[Test]
public void CanCreatePerson()
{
    Person p = new Person("bryan");
    Assert.AreEqual(p.FirstName,"bryan");
    p.FirstName = "Bryan";
    Assert.AreEqual("Bryan",p.FirstName);
}

Trick question. Neither!

There are two ways to improve code coverage -- write more tests, or get rid of code. In this case, I would argue that it better to remove the setter than write any code just to satisfy coverage. (Wow, less really IS more!) Leave the property as read-only until some calling code needs to write to it, at which point the tests for that call site will provide the coverage you need.

"But putting the setter back in is a pain!" -- sure it is. Alternatively, you can leave it in, but make sure you do not write a test for it. If the coverage remains zero for extended periods of time, remove it later. (If you can't remove it because some calling code is writing to it, you missed something in one of your tests.)

Note: In general, plain old value objects like our Person class won't need standalone tests. The exception to this is when you need tests to demonstrate specialized logic in getter/setter methods.

Coverage Tips for Your Project

• Set goals for coverage: Talk to your team about coverage and gather their feedback early in the project. Identify areas that will be difficult to test and develop strategies to make your code more testable. Agree upon a level of acceptable coverage based on your timelines and these constraints. For most projects that start with TDD in mind, 70-80% is very realistic target. I don't have any concrete data to back this up, but I imagine that effort increases by levels of magnitude after a certain percentage.
• Watch for changes in coverage: Rather than looking at overall code coverage percentage as a quality metric, integrate coverage into your build or continuous integration process and look at the change in coverage between builds. Coverage will flutuate as a project matures, eventually it should level out and remain relatively constant between changes. Applaud when it goes up, recognize the hard-work of your team when it stays the same, and investigate when it takes a steep drop. As an added bonus, the integrated coverage logs on your build server can be analyzed over time: it's amazing how developer churn, ramp-up, changes in functionality/design/timelines can become evident in a graphed timeline of failed builds and drops in coverage.
• Use Milestones: Whether you're in an waterfall or agile project, pick milestones in your project where you can look at coverage. I try to fit in at least one code review per iteration and kick them off with a look at code coverage reports ("Yikes! We don't have any tests for this entire namespace, maybe we should fix that.") When coverage is low, I use this time to evangelize the benefits of having tests. Set a goal for next iteration and get buy-in from both the team, management (and client) for well written tests that bump up your coverage. It can be fun motivator for the team.
• Don't Force it:. If you obsess about coverage, you're probably doing it wrong. Deliberately reworking code so that code will light up in the coverage report or writing coverage-serving tests yeilds little benefits -- let it come naturally by writing concise tests. If your tests don't reflect the functionality of the application, fix your tests; if the tests serve only to satisfy coverage they likely don't serve anybody.

TDD Tips: Unit Test Namespace considerations

In my last post, I highlighted some of the test-driven benefits of using the InternalsVisbleTo attribute. In keeping with the trend of TDD posts, here's a recent change in direction I've made about how to separate your tests from your code.

There's a debate and poll going on about where you should put your tests. The poll shows that the majority of developers are putting their code in separate projects (yeaaaa!). Bill Simser's suggestion to have tests reside within the code is a belief that balances dangerously between heresy and pragmaticism. Although I'm opposed to releasing tests with production, one point I can identify with is the overhead of keeping namespaces between your code and your tests in sync. (Sorry Bill, if I wanted my end users to run my tests, I'd give them my Test library and tell them to download NUnit) A long the same lines, at some point our organization picked up some advice that code namespaces should reflect their location in source control. This has proven effective for maintenance as this makes it easier to track down Types when inspecting a stack-trace. Following this advice has led us to adopt a consistent naming strategy for assemblies and their namespaces:

Project	Namespace	Assembly
Component	Company.Component	Company.Component.dll
Test	Company.Component.Test	Company.Component.Test.dll

This works well, but I have a few hang-ups on this design. This strategy pre-dates most of our TDD efforts, and frankly it gets in the way. Here are my issues:

• Namespace Mismatch: We attempt to model the same folder structure between projects and although the folder structure is the same, the namespaces are different. The type Customer would reside in Company.Component.Customer while the CustomerTest would reside in Company.Component.Test.Customer.
• Pure TDD is difficult: When the namespaces are different, it's a lot of extra clicking if you want to create your types as you write your Tests. You have to get out of the Test, create the Type in Library project, switch back to the test and then add the appropriate namespace using statement. If you create the type in the same file as the Test, you'll have to refactor the tests and the Type namespaces when you move it to the library. Most of these issues get caught at compile time, but it's a real nuisance.

However, there is some great advice in the Framework Design Guidelines book which states that assembly names and namespaces don't necessarily have to match. From Brad Abrams site:

Please keep in mind that namespaces are distinct from DLL names. Namespaces represent logical groupings for developers whereas DLLs represent packaging and deployment boundaries. DLLs can contain multiple namespaces for product factoring and other reasons, and namespaces can span multiple DLLs. Since namespace factoring is different than assembly factoring, you should design them independently.

A great example is that there is no System.IO.dll in the .NET framework: System.IO.FileStream resides in MSCorLib.dll while System.IO.FileSystemWatcher resides in System.dll. So if we apply this concept to our solution and think of Tests as a subset of functionality with different packaging purposes, our code and test libraries look like this:

Project	Namespace	Assembly
Component	Company.Component	Company.Component.dll
Test	Company.Component	Company.Component.Test.dll

Here's a snap of my Test Library's project properties...

 

Now that the namespaces are identical between projects, I never have to worry about missed namespace declarations --- I can quickly create Types in the Test library and move them to the library when I'm done. As an added bonus, when I change the namespace using Resharper, it will change my Test library as well. Here's what the TDD flow looks like using Resharper:

1. Write the test, refer to a new non-existent Type.
2. Use Resharper to generate the missing class. The class is created in the same file as the test and is marked internal.
3. Flush out the class using additional tests.
4. When the class is finished, right click the class and choose Refactor -> Move. Specify a new file, the name will automatically reflect the Type name.
5. Drag the new file while holding the SHIFT key from the Test library to the code project. This will physically move the file between the projects and automatically update the project definitions.

Caveats:

• Folder Issues: I should point out that this doesn't solve resolve the folder renaming issue. If you rename the folder in your code library, you'll have to do the same in the Test library. Mind you, Resharper doesn't automatically fix folders when you rename them anyway, so you're going to have to fix this yourself.
• Maintenance Strategy: The maintenance model strategy that allows you to identify the location of a Type in source control based on a stack-trace is partially broken with this design. I say partially because a stack-trace should really only be a concern for production code, and stack-traces for unit-tests don't provide much in the context of a Test Runner. Still, to support troubleshooting, I encourage developers to follow a "Test" naming convention for their tests.
• Intillisense Confusion With your Test and Code library sharing the same namespaces, both TestFixtures and Types will show up in Intellisense when you write code in your Test library. Some might see this as noise when writing tests; others might use it as a good holistic view for classes and associated Tests. If this really bothers you, you could mark your tests with an attribute that would hide the tests from intellisense.

TDD Tips: InternalsVisibleTo - Keep your API clean

...or how to have all the great benefits of clean code and 100% code coverage too.

Although the .NET 2.0 Runtime has been out for quite sometime, I'm still surprised that most people are not aware that the 2.0 framework supports a concept known as "Friend Assemblies", made possible using the InternalsVisibleToAttribute. For me, this handy (and dare i say awesome) attribute solves an age old problem frequently encountered with Test Driven Development and when I first stumbled upon it about two years ago, my jaw hit the floor and I was all nerdy giddy about it.

This has all been blogged about before, but I want to comment on some of the best practices this approach affords us. As a general rule of thumb, you should always try to keep your Unit Tests out of your production code. After all, the classes needed for testing will never be used by end-users, so to prevent bloating up your assembly you should put the tests in a different assembly and leave 'em at home when you release the code. Unfortunately, this produces a strange side-effect: Types and Methods that would normally be marked as internal or private must be made public so that the external Test assembly can access them. You're left with a difficult compromise... either choose to violate your API access rules to support testing, or forgo all unit testing and code coverage for clean code. While the practice of exposing types is relatively harmless, it can introduce some negative side-effects into your project, especially if you're producing a library that is shared with other applications or third parties. Specifically, it can hurt usability and performance:

• Usability of your assembly will be reduced because users will have a full gammet of Types to choose from. A clean API with only a few public facing classes is easier to understand that dozens of utility and helper classes. If you only have a handful of classes, this doesn't apply to you -- but if you've ever inherited a project with hundreds of Types and piss-poor documentation, I know you know what I'm talking about.
• Performance of your API will be compromised if you follow FxCop recommendations -- which btw, is good advise. With all these public facing types you'll need additional parameter-validation and error handling because you can't guarantee how third-parties will access your Types. If your app is for internal-use, you can shirk this responsibility, but be warned: the onus is on you to enforce proper use of your library and to ignore several dozen FxCop violations. If you have third-parties using your library, this is extra plumbing is hard to avoid so it's more likely the Types and Methods are kept private/internal and the tests are simply neglected. Which, IMHO is where you really need the tests since the bugs are more likely to be nested deep in your implementation rather than the public exposed API, and hey ...bugs are bad for business.

Here's a few links that refer to these best practices:

• Validate arguments of public methods
• Validating arguments
• Framework Design Guidelines

Fortunately, the InternalsVisibleTo attribute fixes these issues. By placing the attribute in your assembly, you can keep types as internal and still allow unit testing.

Attribute Usage Examples

Using the attribute is quite simple. The attribute is placed in the assembly that contains the internal classes and methods that you want to expose to other "friend" assemblies. The attribute lists the "friend" assembly.

using System.Runtime.CompilerServices;

[assembly:InternalsVisibleTo("assemblyName")]

MSDN documentation refers to strong names when referring to the friend assemblies, however, a strong-name is not required. This is extremely useful if you're just starting your project or not ready to strong-name the assembly. Note that if you are using a strong-name, it's the full public key and not just the public token.

[assembly:InternalsVisibleTo("assemblyName, PublicKey=fff....")]

To get the full public key of your assembly, you can use the strong name tool that ships with the .NET Framework to extract the public key:

sn -Tp Code.dll

Alternatively, David Kean has published a handy tool that can help you generate the InternalsVisibleTo attribute, so you can simply paste it into your assembly. However, his site is presently being reworked. I have the binary downloaded from his site, though I have no where to host the file. Give me a shout if you're interested... and David, let us know when you're site is back up.

Note: Although the strong-name is optional, you should be using strong-names on your assemblies as a best practice to prevent this type of runtime injection. And if you go down this route, all referenced assemblies must also been signed (all the more reason why you should be using strong-naming in the first place).

A Code Example...

This rudimentary example shows how you can create a class that takes advantage of the InternalsVisibleTo attribute. There are two assemblies: "Code" is my main assembly has the InternalsVisibleTo attribute and public facing API, "Test" is my test library that references "Code". If these assemblies weren't friends, all Types within "Code" would have to be public.

// within Code.dll
[assembly:InternalsVisibleTo("Code.Test")]

namespace Code
{
    public internal class StringUtility
    {
        public static string ProperCase(string input)
        {
            CultureInfo culture = Thread.CurrentThread.CurrentCulture;
            return culture.TextInfo.ToTitleCase(input.ToLower(CultureInfo.InvariantCulture));
        }
    }
}

// within Code.Test.dll
namespace Code.Test
{
    [TestFixture]
    public class StringUtilityTest
    {
        [Test]
        public void CanGetProperCaseFromInternalClass()
        {
            Assert.AreEqual("Hello", StringUtility.ProperCase("HELLO"));
        }
    }
}

Kudos to Rick Strahl for the ProperCase string tip.

The Payoff...

So now that you've got your internal classes with test coverage goodness, treat yourself by opening up FxCop and viewing the reduced violations report.

FxCop before:

This screen capture of FxCop shows a few standard FxCop violations (my assembly isn't strong-named, yet) and a violating public arguments warning.

FxCop after:

Since most FxCop rules are centered around designing public APIs, classes that are marked as internal are exempt from certain rules. This snapshot shows how our internal class isn't subject to requiring additional validation logic.

 

Selenium 0.92 doesn't work in IE over VPN or Dialup

I've been writing user interface tests for my current web project using Selenium. I really dig the fact that it uses JavaScript to manipulate the browser. I'm working on building a Language Pattern, where my unit tests read like a simple domain language -- it involves distilling Selenese output into a set of reusable classes.

I ran into a really frustrating snag during a late-night coding session and I started to freak out a bit -- my Selenium tests just magically stopped working! Instead of getting the Selenium framed window, my site was serving 404 messages. At first I thought the plumbing code that I had written was somehow serving the wrong URL.

I quickly switched my tests to FireFox and was relieved to see them working fine -- and under the same URL. Since my client uses IE 6, dropping Internet Explorer support for UI tests would be a deal breaker. I was surprised to see the tests work when I switched the URL from localhost:80 to localhost:4444, which is the port Selenium's proxy server runs on. The light in my head started to glow...

The aha moment came when I switched back to FireFox: I noticed that none of my FireFox plugins were loaded and that the Proxy server setting had been enabled to route localhost:80 through localhost:4444. Selenium is controlling registry settings for the browser, meaning that some setting was missing in IE. Although Internet Explorer had been configured to use my Selenium proxy-server settings, it ignores these values when on dial-up and VPN connections. You need to specify a different proxy server through an Advanced settings menu.

Both FireFox and Internet explorer use PAC files, which are used to automatically detect the configuration settings for your proxy server. Selenium generates a new PAC file between executions, so you'll quickly find that manually fixing it becomes a pain. To fix, create your own pac file and wire the setting in yourself.

Here's a snap of my connections dialog:

And the contents of my selenium-proxy.pac file:

function FindProxyForURL(url, host) {
        return 'PROXY localhost:4444; DIRECT';
}