The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing

Insights

External Resources

More Info


Page tree
Skip to end of metadata
Go to start of metadata

According to industry experts like Gartner, you will get the most out of your testing automation if you include a healthy mix of emulators, simulators, and real devices as your mobile testing platforms. Why? Because there are aspects of the mobile experience that you can't test on an emulator or simulator, like location-based applications that use manufacture specific device sensors, memory consumption, and CPU usage.

There are a variety of use cases to consider when you're deciding on the mix of emulators/simulators/real devices that you want to use in your testing.

When to Test on Real Devices

When to Test on Emulators/Simulators

  • Need breadth of device types for panel/compatibility testing

  • Need to replicate an issue to match exact model as reported

  • Need pixel perfect display testing

  • Need to test hardware dependencies like CPU, memory, display, GPS, etc.

  • Need to test native ARM Libraries

  • Need to test on a custom OS (for example, Samsung TouchWiz, OnePlus OxygenOS, etc.)

  • Need to test on a native frameworks like Espresso and Robotium
  • Need to test scenarios that require network connectivity (phone calls, send SMS messages, etc.)
  • Need massive concurrency

  • Want to reduce build times

  • Want to save costs

  • Need immediate availability

  • Require extremely low error rates for your test environment

Please choose your preferred platform for your Automated Mobile Application Testing use case:

With Sauce Labs you can run automated tests against many virtual os and platform combinations with Android Emulators and iOS Simulators.

See the following sections for more topics:

What You'll Need

Project Support and Requirements

Before you can upload your application and leverage device emulators and simulators, you must fulfill the project support and requirements. Please review the Automated Mobile Application Testing Admin Guide for further details.

Uploading Your Application to Emulators & Simulators

Before running your test you will need to upload either an Android package file (.apk) for emulators or an iOS package file (.zip) for simulators.

There are three ways you can upload your application for automated testing:

  1. Sauce Application Storage
  2. Install from a Remote Location
  3. Legacy Sauce Storage

For detailed information on how to upload your application please visit Uploading your Application to Emulators and Simulators.

Running Your Test on Emulators & Simulators

After you've uploaded your application to a publicly available source, you must configure your test in order to run automated tests on Sauce Labs. How you configure your application depends on your testing use case:

Appium Tests

Below is some information with regards to running automated Appium tests on Emulators and Simulators.

Setting Appium Capabilities for Emulators and Simulators

Before you run your tests, you will need to configure the Appium capabilities in order to communicate with the Sauce Labs Virtual Devices.

Below are examples of an iPhone 8 project using iOS version 12.2

DesiredCapabilities caps = DesiredCapabilities.iphone();
	caps.setCapability("appiumVersion", "1.13.0");
	caps.setCapability("deviceName","iPhone 8 Simulator");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("platformVersion","12.2");
	caps.setCapability("platformName", "iOS");
	caps.setCapability("browserName", "");
	caps.setCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/iOS.Simulator.SauceLabs.Mobile.Sample.app.2.1.0.zip");
caps['browserName'] = ""
caps['appiumVersion'] = "1.13.0"
caps['deviceName'] = "iPhone 8 Simulator"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "12.2"
caps['platformName'] = "iOS"
caps['app'] = "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/iOS.Simulator.SauceLabs.Mobile.Sample.app.2.1.0.zip"
caps['browserName'] = '';
caps['appiumVersion'] = '1.13.0';
caps['deviceName'] = 'iPhone 8 Simulator';
caps['deviceOrientation'] = 'portrait';
caps['platformVersion'] = '12.2';
caps['platformName'] = 'iOS';
caps['app'] = 'https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/iOS.Simulator.SauceLabs.Mobile.Sample.app.2.1.0.zip';
caps = Selenium::WebDriver::Remote::Capabilities.iphone()
caps['appiumVersion'] = '1.13.0'
caps['deviceName'] = 'iPhone 8 Simulator'
caps['deviceOrientation'] = 'portrait'
caps['platformVersion'] = '12.2'
caps['platformName'] = 'iOS'
caps['browserName'] = ''
caps['app'] = 'https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/iOS.Simulator.SauceLabs.Mobile.Sample.app.2.1.0.zip'
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("appiumVersion", "1.13.0");
	caps.SetCapability("deviceName", "iPhone 8 Simulator");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("platformVersion", "12.2");
	caps.SetCapability("platformName", "iOS");
	caps.SetCapability("browserName", "");
	caps.SetCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/iOS.Simulator.SauceLabs.Mobile.Sample.app.2.1.0.zip");

Below are examples of an Samsung Galaxy S9 Plus project using Android version 8.1

DesiredCapabilities caps = DesiredCapabilities.android();
	caps.setCapability("appiumVersion", "1.9.1");
	caps.setCapability("deviceName","Samsung Galaxy S9 Plus FHD GoogleAPI Emulator");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("browserName", "");
	caps.setCapability("platformVersion","8.1");
	caps.setCapability("platformName","Android");
	caps.setCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/Android.SauceLabs.Mobile.Sample.app.2.2.0.apk");
caps = {}
caps['appiumVersion'] = "1.9.1"
caps['deviceName'] = "Samsung Galaxy S9 Plus FHD GoogleAPI Emulator"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "8.1"
caps['platformName'] = "Android"
caps['app'] = "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/Android.SauceLabs.Mobile.Sample.app.2.2.0.apk"
caps = {};
caps['appiumVersion'] = '1.9.1';
caps['deviceName'] = 'Samsung Galaxy S9 Plus FHD GoogleAPI Emulator';
caps['deviceOrientation'] = 'portrait';
caps['browserName'] = '';
caps['platformVersion'] = '8.1';
caps['platformName'] = 'Android';
caps['app'] = 'https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/Android.SauceLabs.Mobile.Sample.app.2.2.0.apk';
caps = Selenium::WebDriver::Remote::Capabilities.android()
caps['appiumVersion'] = '1.9.1'
caps['deviceName'] = 'Samsung Galaxy S9 Plus FHD GoogleAPI Emulator'
caps['deviceOrientation'] = 'portrait'
caps['browserName'] = ''
caps['platformVersion'] = '8.1'
caps['platformName'] = 'Android'
caps['app'] = 'https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/Android.SauceLabs.Mobile.Sample.app.2.2.0.apk'
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("appiumVersion", "1.9.1");
	caps.SetCapability("deviceName", "Samsung Galaxy S9 Plus FHD GoogleAPI Emulator");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("browserName", "");
	caps.SetCapability("platformVersion", "8.1");
	caps.SetCapability("platformName", "Android");
	caps.SetCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.2.0/Android.SauceLabs.Mobile.Sample.app.2.2.0.apk");

Example Appium Scripts

Below are some links to sample repositories that contain sample scripts to get you started with automated testing on Sauce Labs.

Visit the following repositories for example Appium scripts:

Visit the following repositories for example Appium scripts:

Native Mobile Test Frameworks

Using Native Test Frameworks on Sauce Labs requires the Sauce Runner. Please visit the following pages for further information, example scripts, and demos using the Sauce Runner with native frameworks.

Best Practices and Reporting Test Results

Now that you've been able to get a test running on Sauce, there are a few other modifications you can make to your tests. These example scripts show how to:

You can find more information under Best Practices for Running Tests and Setting Test Status to Pass or Fail.

Additional Topics



The Sauce Labs real device cloud provides you with the ability to run either manual or automated tests across Android and iOS devices, in either a private or public cloud. Some of the features that are specific to our real device cloud include:
  • Manual testing on real devices
  • Private and public real device clouds
  • Support for Appium, Robotium, Espresso, and XCUITest frameworks
  • Support on hundreds of device-OS combinations
  • IPSec VPN secure connections to private clouds
  • Devices cleaning process between sessions to ensure maximum privacy
  • Carrier Network Connectivity (devices with SIM cards)

See the following sections for more information: 

What You'll Need

Before you can upload your application and leverage real devices, you must fulfill the project support and requirements. Please review the Automated Mobile Application Testing Admin Guide for further details.

Uploading and Accessing your Application on Real Devices

Please select your preferred method for uploading your application in order to view the instructions:

  1. Log in to Sauce Labs and select LIVE from the options in the left-hand navigation.
  2. Select Mobile-App.
  3. You will see a list of previously uploaded apps.
  4. To the right of the page, select App Upload to upload a new application (Note: Live Testing on Real Devices only at the moment):

There may be situations where you want to install an application from a downloadable remote location (AWS S3 bucket, a GitHub repository, etc.). The application is completely removed from the real device after the test completes, providing an added layer of security for your application. Please review the following guidelines below before uploading your application:

Please review the following guidelines below before uploading your application:

  1. Make sure your application meets the prerequisite requirements for Android and iOS Mobile Application Testing.
  2. Upload your application to the hosting location.
  3. Ensure Sauce Labs has READ access to the app URL.
  4. In your Appium test script, enter the URL for the application as the "app" desired capability. Below is an example Java snippet:

    Example Java Snippet
    caps.setCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.3.0/Android.SauceLabs.Mobile.Sample.app.2.3.0.apk?raw=true");

Installing your Application on Private Devices

In some cases you may need to upload / install your application to a private device and also prevent the device from broad internet access while under test. The steps to achieve this are:

  • Upload your application to an internal git repository, or private hosting solution with the necessary permissions (e.g. Amazon S3 with a strict bucket policy).
  • Ensure the hosted app URL is available to the machine running the automated test.
  • Ensure that you've enabled the "Require Sauce Connect/VPN" setting in your organization's security settings.


Each Session is a "Fresh" Install

You will not be able to access information about different versions of your application because each session includes a "fresh" installation of your application. 

For Automated Appium Testing Only

You can only install remote applications for automated Appium testing. Espresso and Robotium automated tests are not supported. Live Testing is also not supported.

Application Storage

Here is an example of how to upload an application to Application Storage in the US West Data Center:

Mac OSX / Linux Example
$ curl -F "payload=@/Users/$SAUCE_USERNAME/Downloads/$FILE_NAME.ipa" -F "name=$FILE_NAME.ipa" -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY"  'https://api.us-west-1.saucelabs.com/v1/storage/upload'

For examples of how to access the application in your automated test builds and more detailed information on how to use the Storage API, visit the Application Storage page.


Sauce Temporary Storage

Here is an example of how to upload an application to Sauce Temporary Storage:

MacOS/Linux Example
$ curl -u $SAUCE_USERNAME:$SAUCE_ACCESS_KEY -X POST -H "Content-Type: application/octet-stream" \
"https://saucelabs.com/rest/v1/storage/$SAUCE_USERNAME/$APP_NAME?overwrite=true" --data-binary @path/to/your_file_name

For more examples and more detailed information, visit the Sauce Temporary Storage page.

Running Your Test on Real Devices

Please select your Automated Mobile Application Testing use case:

Below are some examples of how to configure Appium tests for Real Devices.  For real device testing you'll need to grab the project API key and app ID. Please visit the Appium Capabilities for Real Device Testing page for required/optional capabilities and more advanced test configuration options. Also visit the sample test frameworks repository in GitHub for more detailed language-specific examples.

Using Regular Expressions for deviceName

The examples below use regular expressions for setting the deviceName, because a matching device must be present in your account in order for the test to run.

For example:

  • "iPhone.*""iPhone .*" will allocate any iPhone. "
  • .*nexus.*" will allocate any device with the word "nexus" in its display name.
  • "iPhone [67]" or "iPhone [6-7]" will allocate either "iPhone 7" or "iPhone 6" device.
  • "iPhone [67]S" or "iPhone [6-7]S" will allocate either "iPhone 7S" or "iPhone 6S" device. "iPhone 7.*" will allocate "iPhone 7" or "iPhone 7S", or any device that starts with the display name "iPhone 7"

Regular expressions are case insensitive, for example, "iphone .*S" and "IPHONe .*s" are the same.

Using Real Devices on Sauce Labs (app.saucelabs.com)

Below are examples of an iPhone project using iOS version 12.2.

DesiredCapabilities caps = DesiredCapabilities();
	caps.setCapability("username", "SAUCE_USERNAME");
	caps.setCapability("accessKey", "SAUCE_ACCESS_KEY");
	caps.setCapability("deviceName","iPhone .*");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("platformVersion","12.2");
	caps.setCapability("platformName", "iOS");
	caps.setCapability("browserName", "");
	caps.setCapability("app", "storage:filename=<file-name>");
caps = {}
caps['username'] = "SAUCE_USERNAME"
caps['accessKey'] = "SAUCE_ACCESS_KEY"
caps['browserName'] = ""
caps['deviceName'] = "iPhone .*"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "12.2"
caps['platformName'] = "iOS"
caps['app'] = "storage:filename=<file-name>"
caps = {};
caps['username'] = 'SAUCE_USERNAME';
caps['accessKey'] = 'SAUCE_ACCESS_KEY';
caps['browserName'] = '';
caps['deviceName'] = 'iPhone .*';
caps['deviceOrientation'] = 'portrait';
caps['platformVersion'] = '12.2';
caps['platformName'] = 'iOS';
caps['app'] = 'storage:filename=<file-name>';
caps = Selenium::WebDriver::Remote::Capabilities()
caps['username'] = 'SAUCE_USERNAME'
caps['accessKey'] = 'SAUCE_ACCESS_KEY'
caps['deviceName'] = 'iPhone .*'
caps['deviceOrientation'] = 'portrait'
caps['platformVersion'] = '12.2'
caps['platformName'] = 'iOS'
caps['browserName'] = ''
caps['app'] = 'storage:filename=<file-name>'
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("username", "SAUCE_USERNAME");
	caps.SetCapability("accessKey", "SAUCE_ACCESS_KEY");
	caps.SetCapability("deviceName", "iPhone .*");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("platformVersion", "12.2");
	caps.SetCapability("platformName", "iOS");
	caps.SetCapability("browserName", "");
	caps.SetCapability("app", "storage:filename=<file-name>");

Below are examples of an Samsung Galaxy project using Android version 8.1.

DesiredCapabilities caps = DesiredCapabilities();
	caps.setCapability("username", "SAUCE_USERNAME");
	caps.setCapability("accessKey", "SAUCE_ACCESS_KEY");
	caps.setCapability("deviceName","Samsung.*Galaxy.*");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("browserName", "");
	caps.setCapability("platformVersion","8.1");
	caps.setCapability("platformName","Android");
	caps.setCapability("app", "sauce-storage:<upload_filename>");
caps = {}
caps['username'] = "SAUCE_USERNAME"
caps['accessKey'] = "SAUCE_ACCESS_KEY"
caps['deviceName'] = "Samsung.*Galaxy.*"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "8.1"
caps['platformName'] = "Android"
caps['app'] = "sauce-storage:<upload_filename>"
caps = {};
caps['username'] = 'SAUCE_USERNAME';
caps['accessKey'] = 'SAUCE_ACCESS_KEY';
caps['deviceName'] = 'Samsung.*Galaxy.*';
caps['deviceOrientation'] = 'portrait';
caps['browserName'] = '';
caps['platformVersion'] = '8.1';
caps['platformName'] = 'Android';
caps['app'] = 'sauce-storage:<upload_filename>';
caps = Selenium::WebDriver::Remote::Capabilities()
caps['username'] = 'SAUCE_USERNAME'
caps['accessKey'] = 'SAUCE_ACCESS_KEY'
caps['deviceName'] = 'Samsung.*Galaxy.*'
caps['deviceOrientation'] = 'portrait'
caps['browserName'] = ''
caps['platformVersion'] = '8.1'
caps['platformName'] = 'Android'
caps['app'] = 'sauce-storage:<upload_filename>'
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("username", "SAUCE_USERNAME");
	caps.SetCapability("accessKey", "SAUCE_ACCESS_KEY");
	caps.SetCapability("deviceName", "Samsung.*Galaxy.*");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("browserName", "");
	caps.SetCapability("platformVersion", "8.1");
	caps.SetCapability("platformName", "Android");
	caps.SetCapability("app", "sauce-storage:<upload_filename>");

Using the Legacy Platform (app.testobject.com)

Below are examples of an iPhone project using iOS version 12.2

DesiredCapabilities caps = DesiredCapabilities();
	caps.setCapability("testobject_api_key", "project_api_key");
	caps.setCapability("testobject_app_id", "1");
	caps.setCapability("deviceName","iPhone .*");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("platformVersion","12.2");
	caps.setCapability("platformName", "iOS");
	caps.setCapability("browserName", "");
caps['browserName'] = ""
caps['testobject_api_key'] = "project_api_key"
caps['testobject_app_id'] = "1"
caps['deviceName'] = "iPhone .*"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "12.2"
caps['platformName'] = "iOS"
caps['browserName'] = '';
caps['testobject_api_key'] = "project_api_key";
caps['testobject_app_id'] = "1";
caps['deviceName'] = 'iPhone .*';
caps['deviceOrientation'] = 'portrait';
caps['platformVersion'] = '12.2';
caps['platformName'] = 'iOS';
caps = Selenium::WebDriver::Remote::Capabilities()
caps['testobject_api_key'] = 'project_api_key'
caps['testobject_app_id'] = '1'
caps['deviceName'] = 'iPhone .*'
caps['deviceOrientation'] = 'portrait'
caps['platformVersion'] = '12.2'
caps['platformName'] = 'iOS'
caps['browserName'] = ''
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("deviceName", "iPhone .*");
	caps.SetCapability("testobject_api_key", "project_api_key");
	caps.SetCapability("testobject_app_id", "1");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("platformVersion", "12.2");
	caps.SetCapability("platformName", "iOS");
	caps.SetCapability("browserName", "");

Below are examples of an Samsung Galaxy project using Android version 8.1

DesiredCapabilities caps = DesiredCapabilities();
	caps.setCapability("testobject_api_key", "project_api_key");
	caps.setCapability("testobject_app_id", "1");
	caps.setCapability("deviceName","Samsung.*Galaxy.*");
	caps.setCapability("deviceOrientation", "portrait");
	caps.setCapability("browserName", "");
	caps.setCapability("platformVersion","8.1");
	caps.setCapability("platformName","Android");
caps = {}
caps['testobject_api_key'] = "project_api_key"
caps['testobject_app_id'] = "1"
caps['deviceName'] = "Samsung.*Galaxy.*"
caps['deviceOrientation'] = "portrait"
caps['platformVersion'] = "8.1"
caps['platformName'] = "Android"
caps = {};
caps['testobject_api_key'] = 'project_api_key';
caps['testobject_app_id'] = '1';
caps['deviceName'] = 'Samsung.*Galaxy.*';
caps['deviceOrientation'] = 'portrait';
caps['browserName'] = '';
caps['platformVersion'] = '8.1';
caps['platformName'] = 'Android';
caps = Selenium::WebDriver::Remote::Capabilities()
caps['testobject_api_key'] = 'project_api_key'
caps['testobject_app_id'] = '1'
caps['deviceName'] = 'Samsung.*Galaxy.*'
caps['deviceOrientation'] = 'portrait'
caps['browserName'] = ''
caps['platformVersion'] = '8.1'
caps['platformName'] = 'Android'
DesiredCapabilities caps = new DesiredCapabilities();
	caps.SetCapability("testobject_api_key", "project_api_key");
	caps.SetCapability("testobject_app_id", "1");
	caps.SetCapability("deviceName", "Samsung.*Galaxy.*");
	caps.SetCapability("deviceOrientation", "portrait");
	caps.SetCapability("browserName", "");
	caps.SetCapability("platformVersion", "8.1");
	caps.SetCapability("platformName", "Android");

Further Information on Configuring Appium Tests for Real Devices


Using Native Test Frameworks on Sauce Labs requires the Sauce Runner. Please visit the following pages for further information, example scripts, and demos using the Sauce Runner with native frameworks.

Best Practices and Reporting Test Results

Now that you've been able to get a test running on Sauce, there are a few other modifications you can make to your tests. These example scripts show how to:

You can find more information under Best Practices for Running Tests and Setting Test Status to Pass or Fail.

Additional Topics