The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Insights

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

After you've run your tests, you may notice in the Test Results page of Sauce Labs that the test status is Complete, rather than Passed/Failed. On its own, Selenium doesn't have a way to interpret test results, but you can use our API to keep track of passing and failing jobs.

To add a pass/fail status to your Selenium and Appium test results you can implement one of the following solutions:

This logic should be part of your Test Automation Framework, so that the Sauce Labs update_job method will be called appropriately depending on whether the test passes or fails.

Below are some code snippet examples, but please refer to our Sauce Labs Demonstration Code Repositories on GitHub for further information, and more context, on setting the pass/fail status.

Full Example: /selenium-examples/testng/src/test/java/com/yourcompany/Tests/TestBase.java

TestNG Example
    @AfterMethod
    public void tearDown(ITestResult result) {
        getSession().stop(result.isSuccess());
    }

Full Example: /selenium-examples/junit/src/test/java/com/yourcompany/Tests/TestBase.java

JUnit TestWatcher Example
public class SauceTestWatcher extends TestWatcher {
    private SauceSession sauceSession;

    public void setSession(SauceSession session) {
        sauceSession = session;
    }

    protected void succeeded(Description description) {
        sauceSession.stop("passed");
    }

    protected void failed(Description description) {
        sauceSession.stop("failed");
    }
}

Full Example: /selenium-examples/cucumber/src/test/java/com/yourcompany/saucecucumberjvm/StepDefinitions.java

Cucumber Example
    @After
    public void tearDown(Scenario scenario){
        getSession().stop(!scenario.isFailed());
    }

Full Example: /SauceExamples/SeleniumNunit/SimpleExamples/SimpleSauceTest.cs

Nunit Example
        }

        [TearDown]
        public void CleanUpAfterEveryTestMethod()
        {
            var passed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Passed;
            ((IJavaScriptExecutor)_driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed"));

Full Example: /SauceExamples/Selenium4DotNetFramework/Selenium4SauceTests.cs

.Net Example
            //TODO please supply your Sauce Labs user name in an environment variable
            sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
            //TODO please supply your own Sauce Labs access Key in an environment variable
            sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY", EnvironmentVariableTarget.User);
            sauceOptions = new Dictionary<string, object>
            {
                ["username"] = sauceUserName,
                ["accessKey"] = sauceAccessKey

Full Example: selenium-examples/pytest/conftest.py

PyTest example
    sauce_result = "failed" if request.node.rep_call.failed else "passed"
    browser.execute_script("sauce:job-result={}".format(sauce_result))
    browser.quit()

Full Example: /selenium-examples/robotframework/vdc/Tests/resource.robot

Robot Framework Example
End Session
    Run Keyword If  '${TEST_STATUS}'== 'PASS'  Execute Javascript  sauce:job-result=passed
    ...  ELSE  Execute Javascript  sauce:job-result=failed
    Close Browser

Full Example: /on-boarding-modules/mocha-examples/test/module4-mocha-test.js

Mocha-Chai Example
    afterEach(async function() {
        await driver.executeScript("sauce:job-result=" + (this.currentTest.state));
        await driver.quit();
    });

Full Example: demo-js/blob/master/nightwatch/selenium/tests/specs/login.spec.js

Nightwatch Spec Example
    after : function(browser) {
        browser.customSauceLabsEnd();
    },

Full Example: /protractor/tests/configs/protractor.shared.conf.js

Protractor Config Snippet
        /**
         * The test-names need to be updated in Sauce Labs.
         * This needs to be done by adding a new reporter like this. We can determine that
         * we use Sauce Labs if the `sauceUser` and `sauceKey` are filled
         */
        if (processedConfig.sauceUser && processedConfig.sauceKey) {
            jasmine.getEnv().addReporter({
                suiteStarted: async (result) => {
                    await browser.executeScript('sauce:job-name=' + result.fullName);
                },
                specStarted: async (result) => {
                    await browser.executeScript('sauce:context=' + result.fullName);
                },
                specDone: async (result) => {
                    // If there are errors please update the error to Sauce Labs
                    if (result.failedExpectations.length > 0) {
                        const promisses = result.failedExpectations.map(async error => {
                            const errorUpdate = await browser.executeScript('sauce:context=' + error.stack);

                            return errorUpdate;
                        });

                        await Promise.all(promisses);
                    }
                },
            });
        }

Full Example: /selenium-examples/rspec/spec/spec_helper.rb

RSpec Example
  # Ideal implementation is to do a lookup from a YAML file or the like rather than using a switch implementation
  def platform(platform_key)
    case platform_key

Full Example: /selenium-examples/cucumber/features/support/env.rb

Cucumber Example
After do |scenario|
  SauceWhisk::Jobs.change_status(@driver.session_id, scenario.passed?)
  @driver.quit
end