The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Analytics

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

Speedo is a Node JS package that gives users the ability to run performance tests on Sauce directly within their command line. It is an easy way to get started with performance testing and it handles the complexities that are associated with running a front end performance tests and validating performance regressions. Speedo also easily integrates with any CI/CD system, allowing you to plug Sauce Performance into your CI/CD pipeline and start capturing performance within seconds.

Enterprise Plans Only

This feature is available for Enterprise plans only. For more information about other benefits that are included with Enterprise plans, check out our Pricing page.

Chrome Only

This feature is available only for testing web applications with the Google Chrome browser.

See the following sections for more information: 

Speedo Commands

The two main commands that speedo supports can accommodate even the most complex performance test cases:

  • speedo run - Speedo Run allows customers to quickly spin up a performance test on sauce against any pre-defined URL
  • speedo analyze - Speedo Analyze allows customers to validate the performance of URLs that are accessed by one of the previously run WebDriver tests. 

Depending on your testing needs you can use either or both commands in your testing process.

Prerequisites

  • Purchase a Sauce Performance license
  • If installing via NPM, Node.js v8 or later required
  • Code examples assume you have SAUCE_USERNAME and SAUCE_ACCESS_KEY exposed in your environment. If not, you can pass them in as command line parameters -u <username> and -k <access key> or export them via:
$ export SAUCE_USERNAME=Slavocado
$ export SAUCE_ACCESS_KEY=XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\

Watch a Demo

Install Speedo

To get up and running with Speedo, you have the following options:

Install via NPM

Install the package on your system to make the speedo command available.

$ npm install speedo -g
$ speedo --version
> 1.1.1

Install Using Docker

You can use Docker:

$ docker pull saucelabs/speedo
$ docker run saucelabs/speedo run http://json.org -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY
> 1.1.1

Run Command: Test a Single URL

Speedo can be used similarly to other performance testing tools such as Lighthouse or WebPageTest and allows users to run performance tests against a pre-defined URL. Once the command is executed the tool will automatically spin up a series of tests on Sauce Labs against the defined URL and validate the performance.

The run command takes care of the most important components of the test by: 

  • Creating a baseline if you don’t have one for that specific page you are testing
  • Downloading the performance logs for you as test artifact
  • Updating the job status

In addition, you can:

  • Parameterize the run command to change the platform and Chrome version 
  • Specify the metrics you want to analyze

Syntax and Details

$ speedo run <url> 

After the command is finished running, Speedo prints out the results of the performance test. It highlights the metrics you have chosen to validate and prints out any difference to the baseline if a metric outlier is detected.  Depending on whether the test passes or fails, we exit the command with a proper exit code so that your pipeline can potentially block the release of your web application in case a performance regression was introduced.

Identifying a Baseline

If the command is run for the first time, it will run performance tests 10 times in order to establish performance baselines and pre-populate our metrics engine. If the command was run in the past only one job would be executed and newly incoming results would be compared against the baseline data. If the performance values are within baselines, the tests would pass; otherwise it would return with a failure.

Command Defaults

  • Mobile Profile - Tests are executed with a mobile profile that runs the browser with a mobile user agent and throttles CPU and Network to simulate a mobile device. We recommend using this default behavior as it runs the test from a viewpoint where bad performance affects the user experience the most. It also helps us to better determine when a performance regression was introduced.
  • Metrics Validated -  The command only validates for changes in page load times. However, you can easily expand the list of metrics being checked list by using -m flag. For more information, see Configuration Options.

Example

You can kick off a Speedo test by calling:

$ speedo run <URL> // if SAUCE_USERNAME and SAUCE_ACCESS_KEY already exist in the environment
$ speedo run <URL> -u Slavocado -k XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX // if SAUCE_USERNAME and SAUCE_ACCESS_KEY are not accessible or haven't been exported

Analyze Command: Review Existing Tests

While common performance testing tools only allow validating the performance of one specific page URL, Sauce Performance with Speedo allows checking for regressions of any page that can be accessed with some automated steps using tools like Selenium or WebdriverIO. The analyze command accepts the name of the test, as it appears on Sauce Labs, as an attribute and validates the page performance for URLs that the test accessed. For example, a common use case is to validate the performance of the pages that are behind authentication or required to be in a specific state. 

Syntax and Details

$ speedo analyze <test name>

NOTE: The specified test must have the appropriate options set for Performance. 

Command Defaults

  • Metrics Validated -  The command only validates for changes in page load times. However, you can easily expand the list of metrics being checked list by using -m flag. For more information, see Configuration Options.

Example

In the example script below, we put in our login credentials, click on the submit button, and at the end, navigate to the profile page by clicking on the profile icon. In this scenario we would capture the performance of the following pages:

The example script uses WebdriverIO but it can be transferred to any framework that runs on the WebDriver protocol (e.g. Selenium).

Using the analyze command is a 3-step process:

  1. Execute a test on the page. 
  2. Call the analyze command and specify the test from step 1. 
  3. Review the results

Step 1: The Test

Performance Test Code Example
import { remote } from 'webdriverio'

let browser
;(async () => {
    browser = await remote({
        /**
         * authenticate Sauce user
         */
        user: process.env.SAUCE_USERNAME,
        key: process.env.SAUCE_ACCESS_KEY,
        capabilities: {
            browserName: 'chrome',
            platformName: 'Windows 10',
            browserVersion: 'latest',
            /**
             * custom Sauce options
             */
            'sauce:options': {
                /**
                 * to enable Sauce performance for your automated test the
                 * following options need to be provided:
                 */
                extendedDebugging: true,
                capturePerformance: true,
                /**
                 * provide a test name so that you can reference the test
                 * in the Speedo analyze call
                 */
                name: 'Instagram Performance Test'
            }
        }
    })

    /**
     * open url
     */
    await browser.url('https://www.instagram.com/accounts/login')

    /**
     * login
     */
    const username = await browser.$('input[name="username"]')
    await username.setValue('performancetestaccount')
    const password = await browser.$('input[name="password"]')
    await password.setValue('testpass')
    const submitBtn = await browser.$('button[type="submit"]')
    await submitBtn.click()

    /**
     * dismiss notification alert
     */
    const turnOnBtn = await browser.$('button=Turn On')
    await turnOnBtn.click()

    /**
     * open user profile
     */
    const profilIcon = await browser.$('span[aria-label="Profile"]')
    await profilIcon.click()

    await browser.deleteSession()
})().catch(async (e) => {
    console.error(e)
    await browser.deleteSession()
})

Step 2: Run the analyze Command

Once the job above finishes, you will call the analyze command to validate the performance of the two URLs, as follows:

$ speedo analyze “Instagram Performance Test”

Step 3: Review Results

The image above shows that Sauce Labs captured the performance for the following page transitions that happen during the test:

  • Opening the page using the navigate WebDriver command (hard page transition)
  • Transitioning from the login page to the main page after login was successful (hard page transition)
  • View change that opened the user profile page

For more information about the speedo command, see: 


  • No labels