The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Insights

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

Since Selenium does not have a way to interpret tests results on its own, it is important for you to end your test with an assertion that updates the test status from Selenium's Complete to a more informative Passed/Failed status.

What You'll Learn

  • How to use the JavaScript Executor to update the status of your test at the end of the session
  • How to use the Sauce Labs REST API to update the status of your test after it has already completed

See the following sections for more information: 

Update Test Status in Session

You can use the Selenium JavaScript Executor to annotate your test in the @after hook. This is the ideal means of writing your tests to interpret the results as a pass/fail and update the status accordingly.

NOTE: The JavaScript Executer commands can only be run while the test is in session. Once the test has completed, the JavaScript Executor commands are not longer applicable and you must use the REST API to update the test.

Video: Setting Test Status to Pass Fail

Watch this video for a demonstration on using the Selenium JavaScript Executor to annotate your test result with a Passed/Failed status.


Examples

The annotation for calling the JavaScript Executor in your test differs slightly for each framework and language, which are provided in the following code snippet examples. Refer to our Sauce Labs Demonstration Code Repositories on GitHub for further information, and more context, on annotating your tests to record the pass/fail status.

Full Example: /selenium-examples/testng/src/test/java/com/yourcompany/Tests/TestBase.java

TestNG Example
    @AfterMethod
    public void tearDown(ITestResult result) {
        getSession().stop(result.isSuccess());
    }

Full Example: /selenium-examples/junit/src/test/java/com/yourcompany/Tests/TestBase.java

JUnit TestWatcher Example
public class SauceTestWatcher extends TestWatcher {
    private SauceSession sauceSession;

    public void setSession(SauceSession session) {
        sauceSession = session;
    }

    protected void succeeded(Description description) {
        sauceSession.stop("passed");
    }

    protected void failed(Description description) {
        sauceSession.stop("failed");
    }
}

Full Example: /selenium-examples/cucumber/src/test/java/com/yourcompany/saucecucumberjvm/StepDefinitions.java

Cucumber Example
    @After
    public void tearDown(Scenario scenario){
        getSession().stop(!scenario.isFailed());
    }

Full Example: /SauceExamples/SeleniumNunit/SimpleExamples/SimpleSauceTest.cs

Nunit Example
        }

        [TearDown]
        public void CleanUpAfterEveryTestMethod()
        {
            var passed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Passed;
            ((IJavaScriptExecutor)_driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed"));

Full Example: /SauceExamples/Selenium4DotNetFramework/Selenium4SauceTests.cs

.Net Example
            //TODO please supply your Sauce Labs user name in an environment variable
            sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
            //TODO please supply your own Sauce Labs access Key in an environment variable
            sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY", EnvironmentVariableTarget.User);
            sauceOptions = new Dictionary<string, object>
            {
                ["username"] = sauceUserName,
                ["accessKey"] = sauceAccessKey

Full Example: selenium-examples/pytest/conftest.py

PyTest example
    sauce_result = "failed" if request.node.rep_call.failed else "passed"
    browser.execute_script("sauce:job-result={}".format(sauce_result))
    browser.quit()

Full Example: /selenium-examples/robotframework/vdc/Tests/resource.robot

Robot Framework Example
End Session
    Run Keyword If  '${TEST_STATUS}'== 'PASS'  Execute Javascript  sauce:job-result=passed
    ...  ELSE  Execute Javascript  sauce:job-result=failed
    Close Browser

Full Example: /on-boarding-modules/jasmine-examples/spec/module1-jasmine-spec.js

Jasmin Example
            /* Jasmine uses the `expect` method to assert whether or not the test passed */
            expect(title).toBe('Swag Labs');
        });

        /* This sends the results to Sauce Labs.com. */
        driver.executeScript("sauce:job-result=" + (true ? "passed" : "failed"));

Full Example: demo-js/blob/master/nightwatch/selenium/tests/specs/login.spec.js

Nightwatch Spec Example
    after : function(browser) {
        browser.customSauceLabsEnd();
    },

Full Example: /selenium-examples/rspec/spec/spec_helper.rb

RSpec Example
  def build_name
    if ENV['CIRCLE_JOB']
      "#{ENV['CIRCLE_JOB']}: #{ENV['CIRCLE_BUILD_NUM']}"
    elsif ENV['TRAVIS_REPO_SLUG']

Full Example: /selenium-examples/cucumber/features/support/env.rb

Cucumber Example
  options = {browser_name: browser_name,
             platform_name: ENV['PLATFORM_NAME'] || 'Windows 10',
             browser_version: ENV['BROWSER_VERSION'] || 'latest',
             'sauce:options': {name: "#{scenario.feature.name} - #{scenario.name}",


Updating Test Status After Completion

If you did not use the JavaScript Executor to update the status of your test as an assertion in the test code, you can still use the Sauce Labs REST API to update the test status.

What You'll Need

  • Your Sauce Labs account credentials

  • The JOB_ID for the test you wish to update

Call the update_jobs REST API and pass the parameter "passed" with a value of "true" or "false".

Update Test Status
curl PUT -X -u USERNAME:ACCESS_KEY \'https://saucelabs.com/rest/v1/USERNAME/jobs/JOB_ID' \
--header 'Content-Type: application/json' \
--data-raw '{
    "data": {
        "passed": "true"
    }
}'

You can obtain the JOB_ID either by:

  • Collecting and storing the web driver SessionId for the test, which Sauce Labs uses as the JOB_ID
  • Finding the Id value in the test's Metadata tab in the Sauce Labs app, as shown in the following figure