Cool automated user acceptance tests with PhantomJS and .Net

At work, we are constantly releasing new code daily. Unfortunately, during each release, we had to go through the entire site manually just to verify that each page was rendering, looked OK, and was not throwing any errors. So, you can see how this got old pretty quick.

We have decent unit test coverage, but we still need to check the final rendered pages before signing off on a release. So, I decided to build a little test project and PowerShell script that anyone on the team could run on their machine to quickly verify all of the websites pages. Another cool thing it does, is that after all the tests run, it opens a web page of screenshots in your browser so you can see the final results.

Let's get right to the code:

The first thing you need to do is create a new Unit test project in Visual Studio. Then add the Selenium WebDriver Nuget package. Then you need to download the PhantomJS executable. I dropped mine into the root of my project folder, and then set it's properties to copy to output directory.

What is PhantomJS you ask? If you don't know what it is, PhantomJS is a headless webkit-based browser. What it does, is it lets you load up webpages into it, and then interact with them via javascript. It is a really nice tool. If you want to read more about it, check their website out.

First, I created an abstract base class that all of my test classes could derive from.

WebDriverBasedTest.cs

[TestClass]
public abstract class WebDriverBasedTest
{
    protected PhantomJSDriver WebDriver;

    [TestInitialize]
    public void TestInitialize()
    {

        WebDriver = new PhantomJSDriver("path/to/phantomjs/in/bin/folder");
    }

    [TestCleanup]
    public void TestCleanup()
    {
        WebDriver.Quit();
    }

    protected void GoToUrl(string url)
    {
        WebDriver.Navigate().GoToUrl(url);
    }
}

So, now I have my base class. This basically initializes a new PhantomJSDriver on test initialize, and closes it on test cleanup. There is also an easy method to load up pages on each test. Now I can create a new unit test class and start adding tests.

WebTests.cs

[TestClass]
public class WebTests : WebDriverBasedTest
{
    [TestMethod]
    public void page_renders_without_errors()
    {
        this.GoToUrl("http://www.google.com");

        Assert.AreEqual("Google", WebDriver.Title);
    }
}

My tests are done and I can run them. When you run the test, Selenium will launch an instance of PhantomJS using it's built in PhantomJS driver, run the test through there and then check the asserts. With Selenium, you can do all sorts of cool things like selecting elements with CSS, running custom javascript code, etc. I will leave it to you to browse the Selenium docs for code samples.

That's it. Now you can finish your tests. I didn't stop there though. We needed a way to actually see the results of the pages and test runs. So, I set up the code to snag screenshots of each test run. Then after all the tests were run, I created an HTML page of all the results.

First, we need to add some more code to the base class. Add this line to the top of the class:

protected const string testResultsDir = @"c:\path\to\results\dir\";

This is where we will save the html files. Then we add a method to take the screenshots:

protected void SaveScreenshot(string id)
{
    ((ITakesScreenshot)WebDriver).GetScreenshot().SaveAsFile(testResultsDir + id + ".png", ImageFormat.Png);
}

This uses a built in function of Selenium which let's us snag screenshots. Now we can add a method which will let us build up a string to output at the end of our tests.

protected string TestResultBuilder(TestData data)
{
    return string.Format("<div class='test{3}'><h4>{1}</h4>{4}<div class='url'><a href='{2}'>{2}</a></div><a href='{0}.png'><img src='{0}.png' /></a></div>",
        data.Id,
        data.Description,
        data.TestUrl,
        data.Passed ? "" : " failed",
        data.Passed ? "" : "<div class='fail-message'><span>Failed: </span><em>" + data.FailMessage + "</em></div>");
}

Then we need a POCO class to hold our test data for each test:

public class TestData
{
    public string Id { get; set; }
    public string Description { get; set; }
    public string TestUrl { get; set; }

    public bool Passed { get; set; }
    public string FailMessage { get; set; }
}

Pretty simple. This method just builds up an HTML string with the data for a particular test. Now we need to make some modifications to our unit tests class. First we need to add these lines to the top of our class.

private static IList<string> htmlContent = new List<string>();
private TestData testData;

This is what will store our HTML and test data for the class. Now we add a test cleanup method.

[TestCleanup]
public void Cleanup()
{
    this.SaveScreenshot(this.testData.Id);
    htmlContent.Add(TestResultBuilder(this.testData));
}

Now, at the beginning of each test, we setup the data for that test.

this.testData = new TestData {
    Id = "01",
    Description = "http://www.google.com",
    TestUrl = "Google home page renders without error."
};

Now, here is the weird part. When each test finishes, the cleanup method will pass the data for that test to the TestResultBuilder method. That works fine, however, when a test assert fails, we do not have a chance to record the failure and pass that into the TestResultBuilder method. So, we need to wrap our asserts in a try/catch.

try
{
    Assert.AreEqual("Google", WebDriver.Title);
    this.testData.Passed = true;
}
catch (UnitTestAssertException ex)
{
    this.testData.FailMessage = ex.Message;
    this.testData.Passed = false;
    throw new AssertFailedException(ex.Message, ex);
}

This basically captures any assert failures and logs the exception message. However, we still need the Assert to fail for the test runner. So, we throw an AssertFailedException and pass the message along.

So, that's all good. Now, when all the tests in our unit test class run, we need a way to write it all to an html file. So, we need to add some code to the unit test class. We add a class cleanup method.

[ClassCleanup]
public static void ClassCleanup()
{
    WriteResults("Google Tests", String.Join("", htmlContent));
}

The ClassCleanup method is a special method of our unit test runner that will run after all the tests in our class have finished running. Here, we send all of our compiled HTML to a static method on our base class that will dump it all into a file. Add this method to your base class.

protected static void WriteResults(string fileName, string html)
{
    using (var outfile = new StreamWriter(testResultsDir + fileName + "Tests.html"))
    {
        outfile.Write(html);
    }
}

That's it! Each unit test class will create a new file and dump all your test results and screenshots into it. If you are like me, you want to style the HTML output a little so it looks nicer. So, I added a Resource file to my project and added a simple HTML page to it. This way I could add some CSS and make it look nice and pretty. In my HTML file, I added a couple of placeholder text like {{body}}. So, then I can do a quick string replace. Modify your WriteResults method like this.

protected static void WriteResults(string fileName, string html)
{
    var htmlOutput = TestResources.HtmlTemplate;

    htmlOutput = htmlOutput.Replace("{{title}}", fileName);
    htmlOutput = htmlOutput.Replace("{{body}}", html);

    using (var outfile = new StreamWriter(testResultsDir + fileName + "Tests.html"))
    {
        outfile.Write(htmlOutput);
    }
}

I'm done! My tests run OK, and my output looks nice. I ran it like this for quite a while. Then, I got tired of having to open this project each time I needed to run it, and I wanted to make it easier for my team members to run them too.

Scripting it up

Let's create a PowerShell script to do all the annoying work for us.

$testsFile = '.\bin\WebTests.dll'
$resultsFile = 'testresults.txt'

function BuildProject()
{
    Write-Host Building test project...
    & c:\windows\microsoft.net\framework\v4.0.30319\msbuild.exe ".\WebTests.csproj" /nologo /v:q
}

function ClearTestResults()
{
    if (!(Test-Path c:\path\to\results\dir))
    {
        New-Item -ItemType directory -Path c:\tmp -Force
        New-Item -ItemType directory -Path c:\tmp\testResults -Force
    }

    Remove-Item c:\tmp\testResults\*.* -Recurse -Force
}

function RunTests()
{
    Write-Host ' '
    Write-Host Running unit tests...
    Write-Host ' '

    $mstest = 'C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe'

    if (test-path $resultsFile)
    {
        DEL $resultsFile
    }

    & $mstest /testcontainer:$testsFile /detail:errormessage /nologo
}

function LaunchResults()
{
    Remove-Item .\TestResults -Recurse -Force

    Write-Host ' '
    Write-Host Launching test results pages...

    Set-Location -Path C:\tmp\testResults
    Invoke-Item *.html
}

Clear
BuildProject
ClearTestResults
RunTests
LaunchResults
Pause

OK, now we are done. Just run this from within your test project root folder and your off. What this script does is it builds the unit test project. Then it checks for the existance of the results directory where you want your html files to be saved at. If it doesn't exist, it creates it, if it does, it cleans it out. Then it runs the tests and outputs any details of the tests to the console. Then when everything is done, it launches all of the HTML pages with your default web browser. Pretty cool eh?

Well, I hope you enjoyed this post. This has really saved me a lot of time! I can just launch this script and let it run. Then I only need to manually test my newest changes that I am releasing. This takes care of all the regression checking throughout the rest of the site.

Enjoy!