The Sauce Labs Cookbook

Sauce Headless

Front End Performance Testing


External Resources

More Info

Page tree
Skip to end of metadata
Go to start of metadata

Our High Availability Sauce Connect Proxy Setup enables you to run tests using multiple Sauce Connect tunnels and run multiple tunnels grouped together as a tunnel pool, which will be treated as single tunnel. Pools are ideal for running 200 or more parallel tests (high concurrency) because tunnel capacity is limited by a single TCP Connection. 

A major benefit to using the High Availability setup is load balancing; jobs will be distributed among your tunnels. If one of your tunnels goes down, any tests started after that will be routed through another tunnel. That said, if a tunnel instance on your side or a VM tunnel instance on the Sauce Labs side goes down once you've already started running tests, these tests in motion will be impacted.

A strongly recommend best practice is to apply and track tunnels with tunnel identifiers. Otherwise, test traffic initiated using your account will use an unnamed tunnel automatically.  

See the following sections for more information:

What You'll Need

Before getting started with the High Availability setup, our recommendation is to first try the Basic Sauce Connect Proxy Setup to confirm that your system and network architecture is compatible with Sauce Connect.

Basic Sauce Connect Setup Leveraging High Availability

The below diagram shows the Basic Sauce Connect setup leveraging High Availability. On the Sauce Labs side, there are three major components: Sauce Test VM, Sauce Connect Tunnel VM, and Tunnel Pool. The logic flow is:

  1. Test framework makes a request for a new Sauce Test VM, which specifies the tunnel ID of the pool (or uses an unnamed, "default" pool).
  2. A service in Sauce Labs chooses a tunnel from the existing pool at random.
  3. The resulting tunnel from step #2 is assigned to the new test VM.

Multiple Network Routes

In this diagram, we see a setup that allows for multiple network routes when reaching the site(s) under test. There are variations to this, of course. For example, some of the Sauce Connect instances could be in the same network infrastructure as the site(s) under test. You can also imagine leveraging different virtual routes inside the same physical network. What is gained by setting up tunnels in this manner is redundancy for Sauce Connect tests, should one network route fail.

High Availability Tunnel Settings and Commands

You can customize your High Availability tests using these options below. For a full list of High Availability commands and other Sauce Connect options, see Sauce Connect Proxy Command-Line Quick Reference Guide.

Tunnel Pools

Exclusive to our High Availability Sauce Connect Proxy Setup, you can launch multiple tunnels as a tunnel pool that's treated as a single tunnel. Be mindful that each tunnel used in a pool will count toward your tunnel concurrency limit.

Launching Tunnel Pools

Test jobs use tunnel identifiers to distinguish which tunnel or High Availability tunnel pool is going to be used for connection to site under test. You need to specify the desired capability "tunnelIdentifier" in your job configuration. If you use high availability tunnel pools, the same logic is applied. All tunnels in the individual pools need to be started with both  --tunnel-identifier <tunnel_identifier_X> and --no-remove-colliding-tunnels command line options.

What are Colliding Tunnels?

Normally, if you attempt to start multiple tunnels with the same tunnel identifier, only the latest instance of the tunnel with that identifier will stay running. All tunnels with the same identifier started prior to the start of the latest instance will be considered colliding tunnels (tunnels with colliding identifiers) and will shut down.

When creating a tunnel pool, you need to prevent tunnel identifier collision by using Sauce Connect client command line option  --no-remove-colliding-tunnels when starting the tunnels for your tunnel pool. Tunnels will then remain active and tests will be distributed among them. 

Monitoring Tunnel Pools

When running a tunnel pool, we recommend monitoring your activity in Tunnels to ensure your tunnel configuration stability and overall testing efficiency. Here, you can gain insight into all individual tunnels and tunnel pools. You can also check the health of an individual tunnel by running a test on it. 

Code Examples: Launching Sauce Connect Proxy with High Availability

Once you've confirmed that your network is configured to use High Availability, launch Sauce Connect using one of these commands below. You'll need to run this command on each machine where you want to have access to the tunnel pool. 

#On Linux or Mac OSX you would use the following
$ ./sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY --tunnel-identifier myTunnel --no-remove-colliding-tunnels
#On windows you would use the following
> sc.exe -u %SAUCE_USERNAME% -k %SAUCE_ACCESS_KEY% --tunnel-identifier myTunnel --no-remove-colliding-tunnels

Failover and Rolling Restart Functionality

Tunnel pools allow for failover and rolling restart functionality, which improves Sauce Connect Proxy test stability and performance. Test load distribution is balanced automatically and evenly across tunnels using the round-robin load balancing method. This method accelerates test time and allow you to run a high volume of tests in parallel. 

Tunnel pools also abide by general fault tolerance rules. For example, if one tunnel becomes unavailable or shuts down (i.e., due to user shutdown, crash, maintenance, or network partition), you can configure your tunnels to be restored automatically while the test traffic is routed to another tunnel in the pool.

NOTE: Tunnel pools do not self-heal nor restart automatically. If a tunnel stops, you'll need to restart it manually. Other than restarting a stopped or failed tunnel, tunnel pools generally run automatically without user intervention. If you're running a high number of tunnels, you may want to bypass the pool and specify which tunnel to use directly.

When in High Availability mode, we recommend restarting Sauce Connect Proxy tunnels every 24 hours. The tunnel will stay open until all tests associated with it have completed. At the same time, once the shutdown command has been sent, the tunnel is marked inactive in the eyes of the pool, and no new jobs will use it. 

Using Multiple Machines for Failover Functionality

If you're configuring your High Availability Setup with multiple tunnels to provide failover functionality, we recommend setting up each tunnel to run on a separate machine. This way, if a port availability issue or machine failure arises, you will still have active tunnels.

If you're using the same machine for multiple tunnels, you should start Sauce Connect Proxy with unique ports for the Selenium listener and scproxy, and file for the log and pid.