The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Insights

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

Selenium does not have a way to interpret tests results on its own, so it is important for you to end your test with an assertion that updates the test status from Selenium's Complete to a more informative Passed/Failed status.

What You'll Learn

  • How to use the JavaScript Executor to update the status of your test at the end of the session
  • How to use the Sauce Labs REST API to update the status of your test after it has already completed

Update Test Status in Session

You can use the Selenium JavaScript Executor to annotate your test in the @after hook. This is the ideal means of writing your tests to interpret the results as a pass/fail and update the status accordingly.

NOTE: The JavaScript Executer commands can only be run while the test is in session. Once the test is complete, the JavaScript Executor commands are no longer applicable and you must use the REST API to update the test.

Video: Setting Test Status to Pass Fail

Watch this video for a demonstration of using the Selenium JavaScript Executor to annotate your test result with a Passed/Failed status.


Examples

The annotation for calling the JavaScript Executor in your test differs slightly for each framework and language, which are provided in the following code snippet examples. Refer to our Sauce Labs Demonstration Code Repositories on GitHub for further information, and more context, on annotating your tests to record the pass/fail status.

Full Example: /selenium-examples/testng/src/test/java/com/yourcompany/Tests/TestBase.java

TestNG Example
    @AfterMethod
    public void tearDown(ITestResult result) {
        getSession().stop(result.isSuccess());
    }

Full Example: /selenium-examples/junit/src/test/java/com/yourcompany/Tests/TestBase.java

JUnit TestWatcher Example
public class SauceTestWatcher extends TestWatcher {
    private SauceSession sauceSession;

    public void setSession(SauceSession session) {
        sauceSession = session;
    }

    protected void succeeded(Description description) {
        sauceSession.stop("passed");
    }

    protected void failed(Description description) {
        sauceSession.stop("failed");
    }
}

Full Example: /selenium-examples/cucumber/src/test/java/com/yourcompany/saucecucumberjvm/StepDefinitions.java

Cucumber Example
    @After
    public void tearDown(Scenario scenario){
        getSession().stop(!scenario.isFailed());
    }

Full Example: /SauceExamples/SeleniumNunit/SimpleExamples/SimpleSauceTest.cs

Nunit Example
        }

        [TearDown]
        public void CleanUpAfterEveryTestMethod()
        {
            var passed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Passed;
            ((IJavaScriptExecutor)_driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed"));

Full Example: /SauceExamples/Selenium4DotNetFramework/Selenium4SauceTests.cs

.Net Example
            //TODO please supply your Sauce Labs user name in an environment variable
            sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
            //TODO please supply your own Sauce Labs access Key in an environment variable
            sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY", EnvironmentVariableTarget.User);
            sauceOptions = new Dictionary<string, object>
            {
                ["username"] = sauceUserName,
                ["accessKey"] = sauceAccessKey

Full Example: /selenium-examples/pytest/conftest.py

PyTest example
    sauce_result = "failed" if request.node.rep_call.failed else "passed"
    browser.execute_script("sauce:job-result={}".format(sauce_result))
    browser.quit()

Full Example: /selenium-examples/robotframework/vdc/Tests/resource.robot

Robot Framework Example
End Session
    Run Keyword If  '${TEST_STATUS}'== 'PASS'  Execute Javascript  sauce:job-result=passed
    ...  ELSE  Execute Javascript  sauce:job-result=failed
    Close Browser

const {LOGIN_USERS} = require('../configs/e2eConstants');

module.exports = {
    beforeEach: function (browser) {
        browser.page.LoginPage()
            .navigate()
            .waitForDisplayed();
    },

    'Login page: should be able to test loading of login page': function (browser) {
        browser.page.LoginPage().isDisplayed();
    },

    'Login page: should be able to login with a standard user': function (browser) {
        const LoginPage = browser.page.LoginPage();
        const SwagOverviewPage = browser.page.SwagOverviewPage();

        LoginPage.signIn(LOGIN_USERS.STANDARD);
        SwagOverviewPage.waitForDisplayed();
        SwagOverviewPage.isDisplayed();
    },

    'Login page: should not be able to login with a locked user': function (browser) {
        // It doesn't matter which error we check, all errors should be checked in a UT
        // With this UT we just check that A failure is triggered
        const LoginPage = browser.page.LoginPage();

        LoginPage.signIn(LOGIN_USERS.LOCKED);
        LoginPage.containsErrorMessage('Epic sadface: Sorry, this user has been locked out.');
    },

    after : function(browser) {
        // Only update Sauce whe we are testing for Sauce
        if(browser.options.access_key) {
            browser.customSauceLabsEnd();
        }
    },
};

Full Example: /selenium-examples/rspec/spec/spec_helper.rb

RSpec Example
  def build_name
    if ENV['CIRCLE_JOB']
      "#{ENV['CIRCLE_JOB']}: #{ENV['CIRCLE_BUILD_NUM']}"
    elsif ENV['TRAVIS_REPO_SLUG']

Full Example: /selenium-examples/cucumber/features/support/env.rb

Cucumber Example
  options = {browser_name: browser_name,
             platform_name: ENV['PLATFORM_NAME'] || 'Windows 10',
             browser_version: ENV['BROWSER_VERSION'] || 'latest',
             'sauce:options': {name: "#{scenario.feature.name} - #{scenario.name}",


Updating Test Status After Completion

If you did not use the JavaScript Executor to update the status of your test as an assertion in the test code, you can still use the Sauce Labs REST API to update the test status.

What You'll Need

  • Your Sauce Labs account credentials

  • The JOB_ID for the test you wish to update

Call the update_jobs REST API and pass the parameter "passed" with a value of "true" or "false".

Update Test Status
curl PUT -X -u USERNAME:ACCESS_KEY \'https://saucelabs.com/rest/v1/USERNAME/jobs/JOB_ID' \
--header 'Content-Type: application/json' \
--data-raw '{
    "data": {
        "passed": "true"
    }
}'

You can obtain the JOB_ID either by:

  • Collecting and storing the web driver SessionId for the test, which Sauce Labs uses as the JOB_ID
  • Finding the Id value in the test's Metadata tab in the Sauce Labs app, as shown in the following figure