The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Insights

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

The Speedo Node JS package is available with the Sauce Performance license. With it, you can enter a simple command from a command line to measure basic performance of your Sauce app and validate any regressions based on previously executed tests. Speedo also easily integrates with any CI/CD system, allowing you to plug Sauce Performance into your CI/CD pipeline and start capturing performance within seconds.

What You'll Learn

  • Why the Speedo Node JS package is an easy way to get basic performance data about your app
  • What's required to use Speedo
  • How to install Speedo
  • How to format Speedo commands to return performance metrics
  • How to interpret the Speedo results

Getting Started

Here's what you'll find in this section of the Sauce Performance guide: 

What You'll Need

  • Google Chrome (no older than 3 versions from latest) as the test browser
  • Node.js v8 or later (for NPM installations)
  • SAUCE_USERNAME and SAUCE_ACCESS_KEY defined for your environment 

Sample Demo of Speedo Run Command

The following demo shows an example of the input and output of Speedo in terminal.

Install Speedo

You can install the Speedo package for Sauce using NPM or Docker.

NPM Installation

Install the package on your system to make the speedo command available.

$ npm install speedo -g
$ speedo --version
> 1.1.1

Docker Installation

Pull the speedo package and run the installation command using Docker.

$ docker pull saucelabs/speedo
$ docker run saucelabs/speedo run http://json.org -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY
> 1.1.1

Executing the Speedo Commands

Sauce Performance supports the following Speedo commands:

  • speedo run - Get the set of standard performance metrics for any pre-defined URL
  • speedo analyze - Validate the performance of URLs that are accessed by one of the previously run WebDriver tests. 

NOTE: The Speedo commands require your SAUCE_USERNAME and SAUCE_ACCESS_KEY. Export the values into your environment in order to avoid passing them as parameters each time you call a Speedo command. For example:

$ export SAUCE_USERNAME=Slavocado
$ export SAUCE_ACCESS_KEY=XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\

Run Command: Test a Single URL

The Speedo run command initiates a series of tests on Sauce Labs against a single URL defined in the query and returns the corresponding performance results. As part of the execution, the command facilitates the following functions:

  • Establishes a baseline for each metric upon initial instantiation by running a diagnostic 10 times
  • Downloads the performance logs as test artifacts, the local directory of which is provided in the command line output 
Stored performance logs in /var/folders/11/p0wfqdkd4wgct7jdpfzxk4j40000gn/T/tmp-8379w4yCSzRBXqN2
  • Automatically updates the job status to PASS or FAIL based on previously established baseline
  • Outputs reference URLs to the job in the Sauce Labs UI and the Google Lighthouse report
  • Exits with a proper exit code so that your pipeline can potentially block the release of your web application in the event a performance regression was introduced

Run Command Syntax

$ speedo run <url> [params]

Run Command Configuration

You can apply parameter specifications to customize the test execution either by including the parameters as inline arguments or through a config file (speedo.config.js) located in the directory from which Speedo is called. Following is a list of common inline arguments. NOTE: Call $ speedo run -h to view the complete list of run command options.

ParameterDescriptionExample
-n "name"
Identify your performance test with a name.
$ speedo run https://saucelabs.com -n "my_test"
-b "build"
Identify your performance test with a build ID.
$ speedo run https://saucelabs.com -b "build_id"
-u "username"

Pass your Sauce Labs username to authenticate.

$ speedo run https://saucelabs.com -u "SAUCE_USERNAME"
-k "access_key"
Pass your Sauce Labs access key to authenticate.
$ speedo run https://saucelabs.com -k "SAUCE_ACCESS_KEY"
-p "platform"
Specify the platform on which to run the test. If not specified, the default value is Windows 10.
$ speedo run https://saucelabs.com -p "Windows 10"
-v "browser_version"
Specify the version of Chrome (only browser supported at this time) on which to run the test. If not specified, the default value is latest, which resolves to the most current version.
$ speedo run https://saucelabs.com -v "74"
-m "metric"

Tailor the command to measure performance for only certain metrics. See Configuration Options for the list of supported metric values.

If not specified, the test defaults to score, which automatically tests all metrics that currently make up a Lighthouse Performance Score.

$ speedo run https://saucelabs.com -m "firstPaint" -m "domContentLoaded"
-l "log_directory"
Specify a local directory in which to save the performance log file (performance.json). If not specified, the log file is written to a default location in the directory in which you installed Speedo and the path is provided in the output of the call.
$ speedo run https://saucelabs.com -l ./speedo_logs
-t "trace_logs"

Record trace level data in addition to the default log. If specified, the trace.json file is saved to the same folder as the performance log. If not specified, trace level data is not captured.

$ speedo run https://saucelabs.com -t
--throttleNetwork

Tailor the metric measurement standards based on the specific network environment in which the pages are being accessed. See Configuration Options for the list of supported network profile values for this setting.

If not specified, the test defaults to Good 3G.

$ speedo run https://www.saucelabs.com --throttleNetwork online
--throttleCpu

Tailor the metric measurement standards based on the CPU at which the pages are being accessed. See Configuration Options for the list of supported values for this setting.

If not specified, the test defaults to 4 (4X).

$ speedo run https://www.saucelabs.com --throttleCpu 2


Run Command Example

You can kick off a Speedo test by calling:

$ speedo run <URL> // if SAUCE_USERNAME and SAUCE_ACCESS_KEY already exist in the environment
$ speedo run <URL> -u Slavocado -k XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX // if SAUCE_USERNAME and SAUCE_ACCESS_KEY are not accessible or haven't been exported

The output returned for a passing test may look as follows:


Speedo Run Output
$ speedo run https://google.com
✔ Start performance test run with user Slavacado on page https://google.com...
✔ Couldn't find baseline for job with name "Performance test for https://google.com (on "Good 3G" and 4x CPU throttling)", creating baseline...
✔ Run performance test...
✔ Wait for job to finish...
✔ Download performance logs...
✔ Updating job status...

📃 Stored performance logs in /var/folders/ll/p0wfqdkd4wgct7jdpfzxk4j40000gn/T/tmp-8327CMXOiID0imHR

Performance Results
===================
score: 87/100
estimatedInputLatency: 16ms
timeToFirstByte: 584ms
domContentLoaded: 1.7s
firstVisualChange: 1.6s
firstPaint: 1.6s
firstContentfulPaint: 1.6s
firstMeaningfulPaint: 1.6s
lastVisualChange: 2.3s
firstInteractive: 5.2s
load: 3.8s
speedIndex: 2s

Result: pass ✅

⚙️  Runtime settings:

- Network Throttling: Good 3G
- CPU Throttling: 4x
- CPU/Memory Power: 857 (desktop-class device)
- User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 7 Build/MOB30X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3765.0 Safari/537.36

👀 Check out job at https://app.saucelabs.com/performance/dd86a76939d74c7eb67eead18beff442/0
📔 Check out Lighthouse Report at https://api.us-west-1.saucelabs.com/v1/eds/dd86a76939d74c7eb67eead18beff442/performance/C75540FE186687B752EFBD150D7FC0E7/lhr.html


Analyze Command: Review Existing Tests

The Speedo analyze command allows you to compare the performance of multiple URLs that are accessed by a test script performed using automation tool like Selenium or WebdriverIO. Once the test completes, run the analyze command, specifying the name of the test as an inline attribute, to evaluate the page performance for each of the URLs accessed during the test. For example, a Login test would include each of the URL pages associated with successful authentication. 

Analyze Command Syntax

$ speedo analyze "<test name>"

NOTE: The specified test must have the appropriate options set for Performance. 

Analyze Command Configuration

You can apply parameter specifications to customize the test execution either by including the parameters as inline arguments or through a config file (speedo.config.js) located in the directory from which Speedo is called. Following is a list of common inline arguments. NOTE: Call $ speedo analyze -h to view the complete list of run command options.

ParameterDescriptionExample
-u "username"

Pass your Sauce Labs username to authenticate.

$ speedo analyze "Login Test" -u "SAUCE_USERNAME"
-k "access_key"
Pass your Sauce Labs access key to authenticate.
$ speedo analyze "Login Test" -k "SAUCE_ACCESS_KEY"
-r "region"
Identify the Sauce Labs data center for your account. Valid values include us (us-west-1) or eu (eu-central-1). The default value is us.
$ speedo analyze "Login Test" -r "us"
-l "log_directory"
Specify a local directory in which to save the performance log file (performance.json). If not specified, the log file is written to a default location in the directory in which you installed Speedo and the path is provided in the output of the call.
$ speedo analyze "Login Test" -l "/speedo/analyze_log"
-t "trace_logs"

Specify a local directory in which tracing logs for further investigation will be written. If not specified, trace level data will not be captured.

$ speedo analyze "Login Test" -t 
-o "order_index"

The number of pages accessed in the test.

$ speedo analyze "Login Test" -o "2"
-p "page_url"

If only measuring performance for one page in the test, specify the URL of the page to analyze.


$ speedo analyze "Login Test" -p "https://www.instagram.com"


-m "metric"

Tailor the command to measure performance for only certain metrics. See Configuration Options for the list of supported metric values.

If not specified, the test defaults to score, which automatically tests all metrics that currently make up a Lighthouse Performance Score.

$ speedo analyze "Login Test" -m "firstPaint" -m "domContentLoaded"
--allSet the boolean to true to collect performance data for all available metrics (overrides any specified -m settings). By default, this value is false.
$ speedo analyze "Login Test" --all


Analyze Command Example

Before you analyze the performance of page loads in an automation script, make sure you have a named test that has run to completion. 

The following example is based on a WebdriverIO test in which a user logs into Instagram and successfully accesses the home screen.

$ speedo analyze "Instagram Login Test" // if SAUCE_USERNAME and SAUCE_ACCESS_KEY already exist in the environment
$ speedo analyze "Instagram Login Test" -u "Slavocado" -k "XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" // if SAUCE_USERNAME and SAUCE_ACCESS_KEY haven't been exported as environment variables.

The following screenshot shows an example of the returned output for the analysis of the automation test.

The image above shows that Sauce Labs captured the performance for the following page transitions that happen during the test:

  • Opening the page using the navigate WebDriver command (hard page transition)
  • Transitioning from the login page to the main page after login was successful (hard page transition)
  • View change that opened the user profile page

For more information about the speedo command, see: 


  • No labels