Configuring the Selenium Toolkit for Different Environments

Suppose you’ve written some selenium tests using Selenium IDE that target your local machine (http://localhost), but you’d like to repurpose the same tests for your QA, Staging or Production environment as part of your build process.  The Selenium Toolkit for .NET makes this really easy to change the target environment without having to touch your tests.

In this post, I’ll show you how to configure the Selenium Toolkit for the following scenarios:

• Local Selenium RC / Local Web Server
• Local Selenium RC / Remote Web Server
• Remote Selenium RC / Remote Web Server

Running Locally

In this scenario, both the Selenium Remote Control Host (aka Selenium RC / Selenium Server) and the web site you want to test are running on your local machine.  This is perhaps the most common scenario for developers who develop against a local environment, though it also applies to your build server if the web server you want to test is also on that machine. 

Although this configuration is limited to running tests against the current operating system and installed browsers, it provides a few advantages to developers as they can watch the tests execute or set breakpoints on the server to assist in debugging.

This is the default configuration for the Selenium Toolkit for .NET – when the tests execute, the NUnit Addin will automatically start the Selenium RC process and shut it down when the tests complete.

Assuming you have installed the Selenium Toolkit for .NET, the only configuration setting you'll need to provide is the URL of the web site you want to test. In this example, we assume that http://mywebsite is running on the local web server.

<Selenium BrowserUrl="http://mywebsite" />

Running Tests against a Remote Web Server

In this scenario Selenium RC runs local, but web server is on a remote machine.  You’ll still see the browser executing the tests locally, but because the web-server is physically located elsewhere it’s not as easy to debug server-side issues.  From a configuration perspective, this scenario uses the same configuration settings as above, except the URL of the server is not local.

This scenario is typically used in a build server environment.  For example, the build server compiles and deploys the web application to a target machine using rsync and then uses Selenium to validate the deployment using a few functional tests.

Executing Tests in a Remote Environment / Selenium Grid

In this scenario your local test-engine executes your tests against a remote Selenium RC process.  While this could be a server where the selenium RC process is configured to run as a dedicated service, it’s more likely that you would use this configuration for executing your tests against a Selenium Grid.  The Selenium Grid server exposes the same API as the Selenium RC, but it acts as a broker between your tests and multiple operating systems and different browser configurations.

To configure the Selenium Toolkit to use a Selenium Grid, you’ll need to specify the location of the Grid Server and turn off the automatic start/stop feature:

<Selenium
     server="grid-server-name"
     port="4444"  
     BrowserUrl="http://mywebsite.com"
   />
   <runtime
        autoStart="false"
        />
</Selenium>

Announcing Selenium Toolkit for .NET

I’m pleased to announce that I’ve released my first open source project!

The Selenium Toolkit for .NET contains a bundled installer for Selenium libraries (Selenium-RC, Selenium IDE and Selenium client library for .NET) and an NUnit Addin that makes getting started with Selenium easy.

More information can be found here:  http://seleniumtoolkit.codeplex.com/

Selenium Field Notes

My last few projects have leveraged both Selenium-RC and Selenium-Core.  Here's a few notes from the field:

FireFox 3 doesn't work with Selenium 1.0.0 beta 1

When working with Selenium RC out of the box, Selenium stalls when trying to launch an instance of FireFox 3.  The Selenium-RC application works as a browser extension that is marked to only certain versions of the browser, this patch fixes the meta-data for the firefox plugin.

Instructions on how to fix the issue yourself can be found here, and within the comments there's a downloadable selenium-server.jar with the patch already applied.

Use Selenium-RC Judiciously

When you're working with Selenium Remote Control, every selenese command is sent over the network to the java application (even when working locally), so beware of redundant calls.  For example, I wrote several helper methods in my NUnit tests to group common selenese functions together:

public void SetText(string locator, string value) 
{ 
    selenium.Click(locator); 
    selenium.Type(locator,value); 
} 

Since the "Click" event is only needed for a few commands, trimming down the selenese can improve execution speed:

public void SetText(string locator, string value) 
{ 
    SetText(locator, value, false) 
} 
public void SetText(string locator, string value, bool clickBeforeType) 
{ 
    if (clickBeforeType) 
    { 
        selenium.Click(locator,value); 
    } 
    selenium.Type(locator,value); 
} 

Avoid XPath when possible

Using XPath as a location strategy for your elements can be dangerous for long term maintenance for your tests as changes to the markup will undoubtedly break your tests.  Likewise, certain browsers (cough cough IE) have poor XPath engines and are considerably slower (IE is about 16x slower).

Strangely enough, following accessibility guidelines also makes for better functional UI testing.  So instead of XPath locators, consider:

• Use "id" whenever feasible.

  <a id="close" href="#" onclick="javascript:foo();"><img src="close.gif"/></a> 

  selenium.Click("close");

• Use "alt" tags for images.

  <img src="close.gif" alt="close window" onclick="javascript:foo();" />

  selenium.Click("alt=close window");

• Use text inside anchor tags when ids or images are not used.

<a href="/">Home</a> 

selenium.Click("link=Home"); 

Avoid timing code

When working with AJAX or Postback events, page load speed can vary per machine or request.  Rather than putting timing code in your NUnit code (ie Thread.Sleep), take advantage of one of the selenium built-in WaitFor... selenese commands.

To use, you place javascript code in the condition where the last statement is treated as a return value.

// wait 30 seconds until an element is in the DOM
selenium.WaitForCondition("var element = document.getElementById('element'); element;", "3000");

This approach allows your code to be as fast as the browser rather than set to a fixed speed.

Use Experimental Browsers

When testing, I found several cases where I hit security limits, such as uploading a file.  In those cases, you have to use *chrome for Firefox and *iehta for Internet Explorer.  These browser profiles are just like *firefox and *iexplore, except that they run with elevated privileges.

Got a tip?  Leave a comment

Legacy Projects: Test the User Interface with Selenium or WatiN

Following up on the series of posts on Legacy Projects, my legacy project with no tests now has a build server with empty coverage data.  At this point, it's really quite tempting to start refactoring my code, adding in tests as I go, but that approach is slightly ahead of the cart.

Although Tests for the backend code would help, they can't necessarily guarantee that everything will work correctly.  To be fair, the only real guarantee for the backend code would be to write Tests for the existing code and then begin to refactor both Tests and code.  This turns out to be a very time consuming endeavour as you'll end up writing the Tests twice.  In addition, I'm working with the assumption that my code is filled with static methods with tight-coupling which doesn't lend itself well to testing.  I'm going to need a crowbar to fix that, and that'll come later.

It helps to approach the problem by looking at the current manual process as a form of unit testing.  It's worked well up to this point, but because it's done by hand it's a very time consuming process that is prone to error and subjective of the user performing the tests.  The biggest downfall of the current process is that when the going get's tough, we are more likely to miss details.  In his book, Test Driven Development by Example, Kent Beck refers to manual testing as "test as a verb", where we test by evaluating aspects of the system.  What we need to do is turn this into "test as a noun" where the test is a "procedure to evaluate" in an automated fashion.  By automating the process, we eliminate most of the human related problems and save a bundle of time. 

For legacy projects, the best place automation starting point is to test the user interface, which isn't the norm for TDD projects.  In a typical TDD project, user interface testing tends to appear casually late in the project (if it appears at all), often because the site is incomplete and the user interface is a very volatile place;  UI tests are often seen as too brittle.  However, for a legacy project the opposite is true: the site is already up and running and the user interface is relatively stable; it's more likely that any change we make to the backend systems will break the user interface.

There is some debate on the topic of where this testing should take place.  Some organizations, especially those where the Quality Assurance team is separated from the development teams, rely on automated testing suites such as Empirix (recently acquired by Oracle) to perform functional and performance tests.  These are powerful (and expensive) tools, but in my opinion are too late in the development cycle --  you want to catch minor bugs before they are released to QA, otherwise you'll incur an additional bug-fix development cycle.  Ideally, you should integrate UI testing into your build cycle using tools that your development team is familiar with.  And if you can incorporate your QA team into the development cycle to help write the tests, you're more likely to have a successful automated UI testing practice.

Of the user interface testing frameworks that integrate nicely with our build scripts, two favourites come to mind:  Selenium and WaitN.

Using Selenium

Selenium is a java-based powerhouse whose key strengths are platform and browser diversity, and it's extremely scalable.  Like most java-based solutions, it's a hodge-podge of individual components that you cobble together to suit your needs; it may seem really complex, but it's a really smart design.  At its core, Selenium Core is a set of JavaScript files that manipulate the DOM.  The most common element is known as Selenium Remote-Control, which is a server-component that can act as a message-broker/proxy-server/browser-hook that can magically insert the Selenium JavaScript into any site --  it's an insanely-wicked-evil-genius solution to overcoming cross-domain scripting issues.  Because Selenium RC is written in Java, it can live on any machine, which allows you to target Linux, Mac and PC browsers.  The scalability feature is accomplished using Selenium Grid, which is a server-component that can proxy requests to multiple Selenium RC machines -- you simply change your tests to target the URL of the grid server.  Selenium's only Achilles' heel is that SSL support requires some additional effort.

A Selenium test that targets the Selenium RC looks something like this:

[Test]
public void CanPerformSeleniumSearch()
{
    ISelenium browser = new DefaultSelenium("localhost",4444, "*iexplore", "http://www.google.com");
    browser.Start();
    browser.Open("/"); 
    browser.Type("q", "Selenium RC"); 
    browser.Click("btnG");

    string body = browser.GetBodyText();

    Assert.IsTrue(body.Contains("Selenium"));

    browser.Stop(); 
}

The above code instantiates a new session against the Selenium RC service running on port 4444.  You'll have to launch the service from a command prompt, or configure it to run as a service.  There are lots of options.  The best way to get up to speed is to simply follow their tutorial...

Selenium has a FireFox extension, Selenium IDE, that can be used to record browser actions into Selenese.

Using WatiN

WatiN is a .NET port of the java equivalent WatiR.  Although it's currently limited to Internet Explorer on Windows (version 2.0 will target FireFox), it has an easy entry-path and a simple API.

The following WatiN sample is a rehash of the Selenium example.  Confession: both samples are directly from the provided documentation...

[Test]
public void CanPerformWatiNSearch()
{
    using (IE ie = new IE("http://www.google.com"))
    {
        ie.TextField(Find.ByName("q")).TypeText("WatiN");
        ie.Button(Find.ByName("btnG")).Click();

        Assert.IsTrue(ie.ContainsText("WaitN");
    }
}

As WatiN is a browser hook, its API contains exposes the option to tap directly into the browser through Interop.  You may find it considerably more responsive than Selenium because the requests are marshaled via windows calls instead of HTTP commands.  Though there is a caveat to performance: WatiN expects a Single Threaded Apartment model in order to operate, so you may have to adjust your runtime configuration.

WatiN also has a standalone application, WatiN Recorder, that can capture browser activity in C# code.

UI Testing Strategy Tips

Rather than writing an exhaustive set of regression tests, here's my approach:

• Start Small: Begin by writing coarse UI tests that demonstrate simple functionality.  For example, a test that hits the homepage and validates that there aren't any 500 errors.  Writing complex tests that validate specific HTML markup take longer to produce and often tend to be brittle and less maintainable in the long run.
• Map out and test functional areas:  Identify the key functional elements of the site that QA would normally regression test for a build: login, update a profile, add items to a shopping cart, checkout, search, etc.  Some of these will be definite road-blockers that you'll have to work around -- you'll quickly realize you can't guarantee profile-ids and passwords between environments, or maybe your product catalog changes too frequently.  Some will require creative thinking, others may inspire custom testing tools that can perform test-specific queries or functions.  You may even find a missing need in the backend systems that you could build and leverage as part of your tests. 
• Write tests for functional changes:  You don't need to sit down an write an exhaustive site wide regression fixture -- focus on the areas that you touch.  If you write tests before you make any changes you can use these tests to help automate the debugging process.  The development effort is relatively small -- you have to test it anyway a dozen times by hand.
• Write tests for testing bugs!!!:  What better motivation could you have?  This is what regression testing is all about!
• Design for different environments:  The code examples above have URLs hard-coded.  Consider using a tool that uses configuration settings to retrieve or help construct URLs so that you can run your UI tests against your local instance, dev, build-server, QA, integration, etc.  UI Tests make great build-validation utilities!

Selenium 0.92 doesn't work in IE over VPN or Dialup

I've been writing user interface tests for my current web project using Selenium. I really dig the fact that it uses JavaScript to manipulate the browser. I'm working on building a Language Pattern, where my unit tests read like a simple domain language -- it involves distilling Selenese output into a set of reusable classes.

I ran into a really frustrating snag during a late-night coding session and I started to freak out a bit -- my Selenium tests just magically stopped working! Instead of getting the Selenium framed window, my site was serving 404 messages. At first I thought the plumbing code that I had written was somehow serving the wrong URL.

I quickly switched my tests to FireFox and was relieved to see them working fine -- and under the same URL. Since my client uses IE 6, dropping Internet Explorer support for UI tests would be a deal breaker. I was surprised to see the tests work when I switched the URL from localhost:80 to localhost:4444, which is the port Selenium's proxy server runs on. The light in my head started to glow...

The aha moment came when I switched back to FireFox: I noticed that none of my FireFox plugins were loaded and that the Proxy server setting had been enabled to route localhost:80 through localhost:4444. Selenium is controlling registry settings for the browser, meaning that some setting was missing in IE. Although Internet Explorer had been configured to use my Selenium proxy-server settings, it ignores these values when on dial-up and VPN connections. You need to specify a different proxy server through an Advanced settings menu.

Both FireFox and Internet explorer use PAC files, which are used to automatically detect the configuration settings for your proxy server. Selenium generates a new PAC file between executions, so you'll quickly find that manually fixing it becomes a pain. To fix, create your own pac file and wire the setting in yourself.

Here's a snap of my connections dialog:

And the contents of my selenium-proxy.pac file:

function FindProxyForURL(url, host) {
        return 'PROXY localhost:4444; DIRECT';
}