In addition to passing configuration options to Sauce Runner for Real Devices through the command line, you can also create a YAML
configuration file for your tests. The runner will execute based on the test parameters set in the file.
See the following sections for more information:
General Usage
JAVA_HOME=$(/usr/libexec/java_home --version 8) java -jar runner.jar config --path <path to config.yml> --apikey <apikey>
Config Command
When you pass the config
command to the runner, you can no longer use the other configuration options available on the command line (e.g. - devices
, - testname
, etc). The config
command only accepts two parameters: --path <path to config.yml>
and --apikey <apikey>.
Parallel Test Executions
For both examples, the tests can be configured to run in parallel on the Sauce Labs Real Device Cloud.
For each section starting with the - datacenter
directive, a new parallel test thread will spin up for the device indicated. If you specify multiple test classes or test methods, each will be executed serially, in the order presented in the section, on the device. The test results for each device are then rolled into a single set of artifacts (e.g., videos, logs) in the UI. These artifacts may be downloaded via the API after the test suite is complete.
If either example were executed, it would result in four separate test executions in parallel on four different devices. The tests within the section will be assigned to that one device and executed in the order specified.
Example Sauce Runner Configuration File for XCUITest
This snippet includes all the required options for running an XCUITest suite, including the --devices
option to select devices based on both static and dynamic allocation, and the --testsToRun
option to set a specific set of classes/tests to run on a device. The class(es) specified can be written in Swift or Objective-C.
--- # Test framework: "xcuitest" in this example testFramework: xcuitest # Path to the .ipa or .app file (must have the same name as the App created in Sauce Labs) app: ./SampleApp.ipa # Path to the test -Runner app's .ipa or .app file test: ./SampleAppUITests-Runner.ipa # Path where XML test reports will be saved xmlFolder: ./ # A list of devices to be used for this test run devices: # Device 1 example: this will execute every test in the ipa file on a random iOS device # Only specify a DC (either EU or US) - datacenter: EU # Device 2 example: Static Allocation - this test will only run if the named device is available - datacenter: US # Specify the specific device name for static allocation device: iPhone_11_13_5_real_us # set test name (optional) - this is the name that will appear in the Sauce Labs UI (and API results) testname: MyTestName2 # Device 3 example: Dynamic Allocation - datacenter: US # Specify a device name or regex for dynamic allocation: 'iPhone X', 'iPad.*', etc. deviceNameQuery: iPhone 8.* # Platform Version for a dynamic device query. e.g. '13' for all Devices with major version 13 and # arbitrary minor versions or '13.5.1' for a more specific version platformVersion: 13 testname: MyTestName3 # Optional parameters, set to true to enable # phoneOnly: false # tabletOnly: false # privateDevicesOnly: false # if 'true', will run only on Private Devices assigned to your account # Device 4 example: Running subset of tests - datacenter: EU testname: MyTestName4 # Provide a list of test cases or test classes. If you want to run all tests of a class, provide the class name by itself. # To run a specific class method, provide both the class and method names. # You may specify multiple testClass parameters. As described above, each testClass will execute serially on the device indicated. # Each testClass must be preceded by a hyphen (e.g. '- testClass'), whereas testMethod parameters must be at the same indentation # level as testClass, without the hyphen testsToRun: - testClass: SampleTestCase - testClass: SampleTestCase2 testMethod: testItWorks - testClass: SampleTestCase3 testMethod: testThisMethod
Example Sauce Runner Configuration File for Espresso
This snippet includes all the required options for running an Espresso test suite, including the --devices
option to select devices based on both static and dynamic allocation, and the --envs
option to set a specific set of classes/tests to run on a device. The class(es) specified can be written in Java or Kotlin.
--- testFramework: espresso # Path to the app's .apk app: ./SampleApp.apk # Path to the .apk file containing the test bundle test: ./SampleAppUITests-tests.apk # Path where XML test reports will be saved. xmlFolder: ./ # Define a list of devices on which the tests should be executed devices: # Device 1 example # Only specify a DC (either EU or US). This will execute all test methods on the first random Android # device which your account is eligible to use - datacenter: EU # Device 2 example: Static Allocation - this test will only run if the named device is available - datacenter: US # Specify the specific device name for static allocation device: Samsung_Galaxy_S8_real # set test name (optional) - this is the name that will appear in the Sauce Labs UI (and API results) testname: MyTestName3 # Device 3 example: Dynamic Allocation - datacenter: US # Specify a device name or regex for dynamic allocation: 'Samsung Galaxy S7', 'Samsung Galaxy.*', etc. deviceNameQuery: Samsung Galaxy S7 # Platform Version for a dynamic device query. e.g. '7' for all Devices with major version 7 and # arbitrary minor versions or '7.1.2' for a more specific version platformVersion: 7.1 # Optional parameters, set to true to enable # phoneOnly: false # tabletOnly: false # privateDevicesOnly: false # if 'true', will run only on Private Devices assigned to your account testname: MyTestName3 # Device 4 example: Running subset of tests. - datacenter: EU testname: MyTestName4 # Provide a list of test options to espresso # The key-value pairs supported by espresso are documented here: https://developer.android.com/studio/test/command-line#AMOptionsSyntax # In the example below the test would execute all test methods defined in the class com.example.android.TestClassA envs: - key: class value: com.example.android.TestClassA
Uploading Espresso and XCUITest to Storage Before Execution
Example for uploading an iOS test runner:
curl -u "username:APIKEY" -X POST https://app.testobject.com/api/rest -H "Content-Type: application/octet-stream" -H "App-Type: XCUITEST" --data-binary @/path/to/XCUITests-Runner.ipa
Example for uploading an Android test runner:
curl -u "username:APIKEY" -X POST https://app.testobject.com/api/rest -H "Content-Type: application/octet-stream" -H "App-Type: ANDROID_INSTRUMENTATION_TEST" --data-binary @/path/to/androidTest.apk