Best Practices for Running Tests

• If there are are "prerequisite" tasks that need to be taken care of before your test runs, you should include a setup section in your script that executes them before the actual testing begins. For example, you may need to to log in to the application, or dismiss an introductory dialog that pops up before getting into the application functionality that you want to test.

• Similarly, if there are "post requisite" tasks that need to occur, like closing the browser, logging out, or terminating the remote session, you should have a teardown section that takes care of them for you. 

Don't Hard Code Dependencies on External Accounts or Data

Development and testing environments can change significantly in the time between the writing of you test scripts and when they run, especially if you have a standard set of tests that you run as part of your overall testing cycle. For this reason, you should avoid building into your scripts any hard coded dependencies on specific accounts or data, Instead, use API requests to dynamically provide the external inputs you need for your tests. 

What are dependencies? Imagine a test suite with two tests:

@Test
public void testLogin()
{
    // do some stuff to trigger a login
    assertEquals("My Logged In Page", driver.getTitle());
}

@Test
public void testUserOnlyFunctionality()
{
    driver.findElement(By.id("userOnlyButton")).click();
    assertEquals("Result of clicking userOnlyButton", driver.findElement(By.id("some_result")));
} 

<?php
function testLogin()
{
    // do some stuff to trigger a login
    $this->assertEquals("My Logged In Page", $this->title());
}

function testUserOnlyFunctionality()
{
    $this->byId('userOnlyButton')->click();
    $this->assertTextPresent("Result of clicking userOnlyButton");
} 

In both of these examples, testLogin() triggers the browser to log in and asserts that the login was successful. The second test clicks a button on the logged-in page and asserts that a certain result occurred.

This test suite works fine as long as the tests run in order. But second test makes an assumption that you are already logged in, which creates a dependency on the first test. If these tests run at the same time, or if the second one runs before the first one, the browser's cookies will not yet allow Selenium to access the logged-in page, and the second test fails. You can get rid of this dependency by making sure that each test can run independently independently of the others, as shown in these examples.

public void doLogin()
{
	//do some stuff to trigger a login
	assertEquals("My Logged In Page", driver.getTitle());
}
 
@Test
public void testLogin()
{
	this.doLogin();
}
 
@Test
public void testUserOnlyFunctionality()
{
	this.doLogin();
	driver.findElement(By.id("userOnlyButton")).click();
	assertEquals("Result of clicking userOnlyButton", driver.findElement(By.id("some_result")));
}

<?php
function doLogin()
{
    // do some stuff to trigger a login
    $this->assertEquals("My Logged In Page", $this->title());
}

function testLogin()
{
    $this->doLogin();
}

function testUserOnlyFunctionality()
{
    $this->doLogin();
    $this->byId('userOnlyButton')->click();
    $this->assertTextPresent("Result of clicking userOnlyButton");
} 

The main point is that it is dangerous to assume any state when developing tests for your app. Instead, you should find ways to quickly generate desired states for individual tests. In the example, this is accomplished with the doLogin() function, which generates a logged-in state instead of assuming it. You might even want to develop an API for the development and test versions of your app that provides URL shortcuts that generate common states. For example, a URL that's only available in test that creates a random user account and logs it in automatically.

Instead, use sensible semantics for CSS IDs and form element names, and try to restrict yourself to using these semantic identifiers. For example, in Java you could designate elements with driver.findElement(By.id("someId")); or driver.findElement(By.name("someName")); or, in the example of PHP, you could use $this->byId() or $this->byName() . This makes it much less likely that you'll inadvertently break your page by shuffling around some lines of code.

DesiredCapabilities caps = DesiredCapabilities.firefox();
caps.setCapability("platform", "Windows XP");
caps.setCapability("version", "37.0");
caps.setCapability("name", "Web Driver demo Test");
caps.setCapability("tags", "Tag1");
caps.setCapability("build", "build-1234");
WebDriver driver = new RemoteWebDriver(
new URL("http://YOUR_USERNAME:YOUR_ACCESS_KEY@ondemand.saucelabs.com:80/wd/hub"),
    caps);

desired_cap = {
    'platform': "Mac OS X 10.9",
    'browserName': "chrome",
    'version': "31",
    'build': "build-1234"
} 

desired_cap = {
    'platform': "Mac OS X 10.9",
    'browserName': "chrome",
    'version': "31",
    'build': "build-1234",
    'tags': [ "tag1", "tag2", "tag3" ]
}

NOTE: While it is technically possible to use the same build name for multiple runs, this will cause all results in Sauce Labs to incorrectly appear as part of a single run, which will cause your test results for those builds to be inaccurate. Sauce Labs recommends using a time stamp or a CI job/build number at the end of your build tag in order to differentiate between builds, for example:

caps.setCapability("build", "build: " + System.getenv("BUILD_TAG"));

caps.SetCapability("build", "build: " + System.Environment.getEnvironmentVariable("BUILD_TAG"));

build: 'build: ' + process.env.BUILD_TAG,

'build': 'build: ' + os.environ["BUILD_TAG"],

build: 'build: ' + ENV['BUILD_TAG'],

There are many situations in which your test script may run ahead of the website or application you're testing, resulting in timeouts and a failing test. For example, you may have a dynamic content element that, after a user clicks on it, a loading appears for five seconds. If your script isn't written in such a way as to account for that five second load time, it may fail because the next interactive element isn't available yet. 

The general advice from the Selenium community on how to handle this is to use explicit waits. While you could also use implicit waits, an implicit wait only waits for the appearance of certain elements on the page, while an explicit wait can be set to wait for broader conditions. Selenium guru Dave Haeffner provides an excellent example of why you should use explicit waits on his Elemental Selenium blog. Whether you use explicit or implicit waits, you should not mix the two types in the same test. 

These code samples, from the SeleniumHQ documentation on explicit and implicit waits, shows how you would use an explicit wait. In their words, this sample shows how you would use an explicit wait that "waits up to 10 seconds before throwing a TimeoutException, or, if it finds the element, will return it in 0 - 10 seconds. WebDriverWait by default calls the ExpectedCondition every 500 milliseconds until it returns successfully. A successful return for ExpectedCondition type is Boolean return true, or a not null return value for all other ExpectedCondition types."

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

ff = webdriver.Firefox()
ff.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(ff, 10).until(EC.presence_of_element_located((By.ID, "myDynamicElement")))
finally:
    ff.quit() 

WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));

IWebDriver driver = new FirefoxDriver();
driver.Url = "http://somedomain/url_that_delays_loading";
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
IWebElement myDynamicElement = wait.Until<IWebElement>((d) =>
    {
        return d.FindElement(By.Id("someDynamicElement"));
    });

require 'selenium-webdriver'

driver = Selenium::WebDriver.for :firefox
driver.get "http://somedomain/url_that_delays_loading"

wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds
begin
  element = wait.until { driver.find_element(:id => "some-dynamic-element") }
ensure
  driver.quit
end

Small

Small refers to the idea that your tests should be short and succinct. If you have a test suite of 100 tests running concurrently on 100 VMs, then the time it will take to run the entire suite will be determined by the longest/slowest test case. Keeping your tests small ensures that your suite will run efficiently and provide you with results faster. 

Atomic

An atomic test is one that focuses on testing a single feature, and which makes clear exactly what it is that you're testing. If the test fails, then you should also have a very clear idea of what needs to be fixed. 

Autonomous

An autonomous test is one that runs completely independently of other tests, and is not dependent on the results of one test to run successfully. In addition, an autonomous test should use its own data to test against, and not create potential conflicts with other tests over the same data. 

• The SeleniumHQ documentation of page objects, hosted on Google Code
• The documentation for the Intern testing framework provides a good explanation of page objects and an example in JavaScript
• The cheezy/page-object GitHub repository includes the page-object gem for Ruby, as well as a good tutorial on how to create page objects

sauce: break

sauce: break is a JavaScript statement that you can insert into your Selenium execute_script command to both identify and interrupt flaky tests for further diagnosis. You can find more information in the topics Annotating Tests with Selenium's JavaScript Executor and Live Testing on Virtual Mobile Devices.

The Breakpoint Button

When your test is running, you can use the Breakpoint button on the Test Details page to interrupt the test and assume manual control of the browser. When you're done investigating, click Stop Job, and the test that you breakpointed will be marked as such on the Test Details page. 

Reusing an account between test runs can lead to:

• Problems with account state before testing starts
• Failures when account setup code has changed
• Failures that only show on other accounts (like your production customers)
• Parallelisation problems between tests using the same account

When should you create a new account?

Roughly speaking, whenever your tests aren't interacting with previous or future tests:

• Running a test on different platforms
• Running a test that doesn't depend on other test state (tests with shared state are also an antipattern)
• Every time you run a test suite

Solution - Don't use real credentials

The best way to avoid this is to avoid using "real" credentials in tests, through the creation of temporary accounts.

Workaround - Transmit session tokens only

You can also avoid sensitive credentials using Selenium's ability to extract and inject cookies into accounts:

1. Create a session in your environment, either directly in the application engine, or by using a local Selenium session or headless browser.
2. Extract the session tokens (local storage objects, credentials, cookies, etc.).
3. Use Selenium to push these objects and tokens into the browser under Sauce Labs' control

This technique avoids sending plain text passwords, however, the sent tokens and cookies are still logged. If your session tokens are not time-sensitive, this provides only security through obscurity.  We recommend using time-sensitive session tokens.

Workaround - Change passwords after tests

If generating tokens and using unique temporary accounts is not possible, we recommend you have test actions your suite always takes, in order to change to a new, randomly generated password.  

After each test, use a locally automated browser, a direct connection to your application database or a headless browser to change your test account's password to a new, randomly generated password.  Ensure this password is stored in your CI environment, a credential store, or some other method.

In order to prevent credential loss from blocking test suites, you may want to start each test suite by changing the password, again, either by using a headless browser or local Selenium session to perform your password recovery process, or by directly interacting with your application's database.

• Imperative testing or programming is essentially spelling out with as much detail as necessary how to accomplish something.
• Declarative testing or programming is only specifying (or declaring) what needs to be accomplished.

This is seen acutely in BDD circles because the goal of BDD is to get all of the interested parties (Project, Dev, Test, Business, etc) to collaborate on the requirements of a feature before anyone begins working on the implementation. Many testers have latched on to BDD tools as glorified test runners rather than a way to actually facilitate BDD practices. This results in features that include actual code and data structures. Less problematic, but still usually missing the point, is a heavy reliance on imperative scenarios. For example: 

1. Given I open a browser
2. And I navigate to http://example.com/login
3. When I type in the username field bob97
4. And I type in the password field F1d0
5. And I click on Submit button
6. Then I should see the message Welcome Back Bob

This scenario is not focused solely on the business requirements, and actually needs to have knowledge implementation specific details in order to work. The fact that the user’s username is bob97 has nothing to do with the business requirements of the company. If BDD features are designed to represent the business logic, then they should only be changed if the business requirements change. If bob97 changed his password to 1<3MyD0g, the page location changes ,or the success message changes, this test would fail, even though the business needs are exactly the same.

A declarative example of the same functionality looks like this:

1. Given I am on the Login Page
2. When I sign in with correct credentials
3. Then I should see a welcome message

This is all information that the business cares about, is easier to read, and leaves it to the implementation to specify how a successful login is accomplished.

This principle can be applied to any language or test runner. Tests should largely focus on what needs to be accomplished, not the details of how it is done. They should mostly be understandable when read by non-developers. This approach goes very well with using the Page Object Pattern. Keep the business logic in the test, and put all of the information about the drivers, the element locators, the timing, etc in the Page Object.  

How Does Maven Manage Dependencies?

You add dependencies for your project to your Maven configuration file (also known as the  pom.xml file, for Project Object Model). As you build your project using Maven, it resolves these dependencies and downloads the dependencies to your local repository folder. This folder is usually located in your user’s home folder and is named .m2. Each dependency downloaded from the repository is a project itself, and has its own dependencies. Maven recursively resolves all of these dependencies for you, and then merges shared dependencies and downloads them.  At the end of the process you end up with a list of dependencies that are needed to run your project on your local machine. For full details about how this process works, check out the Maven documentation.

How Do I Get the Dependencies for a Project?

From this brief description of how Maven dependencies work, you may notice a problem: How do you know exactly what dependencies are required for a project, and if you don’t have Internet access or are trying to run your project offline, how do you make sure you have all the dependencies you need available locally? Fortunately, Maven includes several commands that you can use to make sure you have all the dependencies and repositories set up so that your project will build and run with no errors.

1. First, check for version updates to your dependencies, and then update the outdated dependencies in your pom.xml file as necessary.
   $ mvn versions:display-dependency-updates

2. Get a list of your repositories, and make sure they’re pointing to all the correct dependencies.
   $ mvn dependency:list-repositories

3. Get a list of your plugin and project dependencies, and make sure they’re all available in your private or local repository. 
   $ mvn dependency:go-offline

If you want to automate the process, or just get a cleaner output of your dependencies, you can also use this bash command:

$ mvn -o dependency:go-offline|grep ":*.jar"|awk '{split($0,a,":");print a[2]}'