Skip to end of metadata
Go to start of metadata

If you happen to have a large number of tests and access to a hosted Selenium grid, or one of your own creation, you are probably wondering how you could make better use of it. Today even a conventional laptop computer can have 2 to 4 processor cores (or even more), and running a single task, or test, at a time would only use fraction of the computing power available. To really get the most out of the power and capacity available to you, you would want to run your tests across multiple browsers at the same time, or go even further and distribute all your tests across all available browsers.

In this topic I’ll explain how to speed up test execution. The examples I will present will be primarily in Java, however the principles of multithreaded programming and basic concepts of object-oriented programming that I’ll go over should be useful to everyone regardless of your preferred programming language.

Terminology

There are essentially two types of tasks that your processor runs, each of which utilizes virtual memory in a different way.

 

Heap

Virtual memory space designated for a process in memory.

Process

A task running on a processor that owns its own private heap. This provides better data protection, but effectively has more overhead due to heap space not being shared. Build tools or libraries that refer to the parallel test instances as processes and forks would spawn processes for these tests.

Thread

A task running within the confines of a process, which shares the heap space of the parent process and may share parts of the heap with other threads. This has poor data protection, but has less overhead. Build tools and libraries refer to threads simply as threads.

 

As these descriptions show, both processes and threads have their advantages and disadvantages relative to data security and virtual memory consumption. Depending on your test framework, test organization, and the build tools you’re using, you may want to use a combination of both types of tasks. In some cases, build tools or test frameworks will be using both types of tasks behind the scenes to help you with your build and test efforts.

Toolchain

These are some basic components for setting up your testing toolchain in Java.

 

Language

Java (v7+)

Test Runner

TestNg (v6.9.+)

Build Tool

Maven (v3.+)

Plugins

Maven Surefire Plugin (v2.18+)

 

Preparing Tests for Parallel Execution

There are a few key points to pay attention to when designing tests for parallel execution, but are also best practices for any type of testing setup.

Your tests should be:

  • Independent: The test should not depend on anything other than the defined setup and teardown methods, nor the order of execution
     

  • Concise: Test one feature at time and have your tests fail with intelligible error messages, so you can isolate failing components or features correctly
     

  • Repeatable: Your test should return the same result on the same version of the target application or website, using the same set of test parameters. In other words, your tests should not be susceptible to issues due to network delays or server performance.
     

  • Reproducible: Your test should returning the same result on the same version of the target application or website using a different set of test parameters. For example, changing the browser, browser version, or host operating system should not produce different test results.

These best practices are often very challenging to follow, as they place a great burden on the test framework to set up concise tests that are repeatable, reproducible and independent. The Sauce Labs documentation wiki includes examples of how to set up tests that follow these best practices, along with additional tips.

Configuring Maven Surefire for Parallel Test Runs

In an effort to keep our discussion of parallel testing concise and generally applicable, we will be limiting it to the usage of Maven as our build system and the Surefire plugin as our test executions agent.

Base pom.xml File

<!--- Snippet from pom.xml file of your maven project -->
<project> 
 ... 
 <build> 
   <!-- To define the plugin version in your parent POM --> 
   <pluginManagement> 
     <plugins> 
       <plugin> 
         <groupId>org.apache.maven.plugins</groupId> 
         <artifactId>maven-surefire-plugin</artifactId> 
         <version>2.19.1</version> 
       </plugin> 
       ... 
     </plugins> 
   </pluginManagement> 
   <!-- To use the plugin goals in your POM or parent POM --> 
  </plugins> 
   [...] 
     <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-surefire-plugin</artifactId> 
       <version>2.19.1</version> 
        <configuration> 
         [...] 
       </configuration> 
     </plugin> 
   [...] 
</plugins>
 </build>

 ... 
</project>
<!--- Snippet from pom.xml file of your maven project --> 

Option 1: Using Configuration Properties in the pom.xml File

In the base pom.xml file example, the <configuration> section is highlighted. This is the section we will use to set up Surefire for parallelization of threads.

In this example of setting the <configuration> options, the maximum total thread count is set at 18. This is the the dataproviderthreadcount (6) multiplied by  the number of allowable threads per test method, which is set by the threadCount property (3). If you don’t provide a value for dataproviderthreadcount, it assumed to be 1.

This configuration will execute tests defined in the default testng.xml test suite in parallel.

Example of Configuration Properties for Setting Thread Count for Parallel Tests

<configuration>
 <properties>
   <property>
     <name>parallel</name>
     <value>methods</value>
   </property>
   <property>
     <name>threadCount</name>
     <value>3</value>
   </property>
   <property>
     <name>dataproviderthreadcount</name>
     <value>6</value>
   </property>
 </properties>
</configuration> 

Option 2: Setting Configuration Properties to Use a Test Suite Configuration File

In this example, the configuration properties aren’t set individually, but refer to a test suite configuration file in which the properties are set.

The properties being set are the same, however any properties that are set in the configuration file will be overridden by their values in the suite definition file. For this reason, it’s best to either define all your configuration properties in the pom.xml file, or in the test suite file, but not in both places.

Example of Setting Configuration Properties to Use a Configuration File

<configuration>
 <suiteXmlFiles>
   <suiteXmlFile>testSuite.xml</suiteXmlFile>
   <suiteXmlFile>testSuite2.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>

Using a test suite configuration file in conjunction with the configuration properties will produce the exact same operational behavior as option 1, but gives you the increased flexibility to customize settings suite by suite.

In the configuration file, the parallel option gives you the option to parallelize by individual test methods, by tests defined in the test name attribute, and by suites defined by the “ suite name ” attribute. For more information on using the “ parallel ” property, check out the Maven Surefire documentation.

Example of a Test Configuration File

<?xml version="1.0" encoding="UTF-8"?> 
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> 
<suite  name="Suite"   parallel = "methods" thread-count="3" data-provider-thread-count="6"> 
   <test name="test1"> 
       <packages> 
           <package name="com.yourcompany.yourapp"></package> 
       </packages> 
   </test> <!-- Test --> 
</suite> <!-- Suite -->

Option 3: Using Maven Surefire Forked Test Execution

The <forkCount> property shown in this example lets you to run each of your test classes in a separate process. An important point to keep in mind using this option is that the JVM processes in use will have higher overhead than the thread based parallelization options that run in a single JVM. On the other hand, this option provides complete isolation of tests as far as memory resources are concerned, and is generally less problematic in situations where hardware resources are not a concern, for example a small number of parallel tests running on a single host computer. Check out the Surefire documentation for more information.

Example of Using <forkCount> as a Configuration Option

<configuration>
   <forkCount>3</forkCount>
</configuration>

Any of these options can be used in conjunction with each other to configure Surefire for parallel testing.

Configuring Test Code for Parallel Test Runs

Making your Test Classes Thread Safe

When you parallelize your test runs by test method, you will need to make sure that shared resources within your test classes are isolated within each thread. You can do this by initializing and keeping all related resources within the test method, or by simply utilizing readily available Java libraries, which is the better solution, and will help keep your test suites clean and organized. Keeping all your resources within the test method,  while functional, will make your code a lot harder to read and maintain, and will make code reuse near impossible.

Webdriver Generation Snippet from Test Class

...
private ThreadLocal<WebDriver> webDriver = new ThreadLocal<WebDriver>();
...
protected void createDriver(String browser, String version, String os, String methodName) 
           throws MalformedURLException, UnexpectedException { 
        ...
       this.webDriver.set(new RemoteWebDriver( 
               new URL("http://" + authentication.getUsername() + ":" + authentication.getAccessKey() +
                        seleniumURI +"/wd/hub"), capabilities)); 
        ...
   }
...

In addition to keeping test specific resources thread local, another best practice to keep in mind is reviewing all of your static class members, and only keeping the ones that are truly intended to be static members. This applies to all of your classes, including the test classes, page objects, and anything else that may be loaded during the test execution.

Configuring Your Data Provider for Parallel Runs

Data Provider Definition from Test Class

@DataProvider(name = "hardCodedBrowsers", parallel = true ) 
   public static Object[][] sauceBrowserDataProvider(Method testMethod) { 
       return new Object[][]{ 
               new Object[]{"internet explorer", "11", "Windows 8.1"}, 
               new Object[]{"chrome", "41", "Windows XP"}, 
               new Object[]{"safari", "7", "OS X 10.9"}, 
               new Object[]{"firefox", "35", "Windows 7"}, 
               new Object[]{"opera", "12.12", "Windows 7"},
               ... 
       }; 
   } 

By passing in parallel = true to the  to the @DataProvider annotation, you enable the parallel execution of tests using the data provider. Combined with the Maven Surefire property dataproviderthreadcount this parameter lets you run your test methods in parallel using items from the data provider list.

What Happens at Runtime?

With these configurations in place your test would be running in parallel in batches of 18 tests at a time, and upon completion of each batch a new batch will be scheduled. All you would need to do is to invoke the Maven test goal using the command line mvn test.

At this point what you should  run a few experiments, and monitor your resource usage and test performance to make sure your tests can scale to the degree of parallelism you aim for, and you can get the most of your test resources.

A sample project demonstrating steps explained in this article is available here .

If you have difficulty getting your tests to run as expected, check out Things That Can Go Wrong with Running Java Tests in Parallel, and How to Debug Them.

  • No labels