One Test for Every Environment
In an ideal world, you want your test scripts to be agnostic about the environment you run them in, whether that's on a local machine, or from a continuous integration server. If you hard-code configuration parameters into your tests, however, you can easily create situations in which a test will run in one environment, but not the other, unless you change the configuration every time you run them for the specific environment where they're running.
Imagine, for example, that you are testing from behind a firewall and use Sauce Connect to create a tunnel to the application you want to test. You have tests that you use as part of your continuous integration builds, and that you also execute from a local machine. The CI server uses a Sauce Connect tunnel that is constantly up, and which is associated with your Sauce Labs account. This tunnel has a unique identifier, like
ci_server. You also have a Sauce Connect tunnel that you run from your local machine, which has a different identifier, like
dev_tunnel. If you hardcode one tunnel identifier into your tests, there's a chance that your local machine could use the tunnel for your CI server, or the CI server could try to use your personal Sauce Connect tunnel, and in both cases, your tests could fail because the tunnel is down, or because the CI server is consuming all the network bandwidth.
One solution would be to maintain two sets of identical tests, one for the CI server, one on your local machine. A better solution would be to set up your tests to use environment variables that will configure them based on the environment where they're running. This means that the behavior of your tests will change based on the environment, without you having to make any code changes.
The catch with environment variables that change the behavior of your tests is that you may have a specific default behavior you always want your test to follow. The solution is to set up your tests to use the environment variables in a "special" environment, like with the CI server, or only in production, and otherwise use the default. The way to achieve this is have your tests only use the environment variable when it has been set to some value. You only need to set this value once for the CI and production environments, and if the test runs in development or locally without the value, it will simply default to the basic behavior.
Check for Values Rather than Flags
It's important that your test check for actual values set in the environment variables, rather than just the presence of the value. That's because it's possible for the environment variable to to be present, but contain an empty string. For example, if your test only checks for the presence of the variables, all three of these variables would be considered "true:"
It's also possible that the values for these variables could be something other than "yes"/"no", true/false:
Checking for the actual value of the variable is more robust, and will help ensure that your tests are behaving the way you expect them to in any of the environments where they run.
Basic Steps for Setting Up Your Tests to Use Environment Variables
- Decide what the default behaviour should be.
- Set an environment variable with a clear name in the environment where the default does NOT apply, like
- Update your test scripts to check the value of that environment variable and, if it matches, change the behaviour of the test script.
- Make sure that the default behavior occurs if the environment variable is missing or has the wrong value.
A Default or Another Value
In the example of using one Sauce Connect tunnel for a CI server, and another for local development, you want to set up your CI server with an environment variable for its dedicated Sauce Connect tunnel, and then have your test check for the value of any
tunnel_identifier environment variables. If it finds one, it will use the value for that variable, but if the variable is absent or an empty string, it will default to using the tunnel for local development.
Because you could have multiple named tunnels, setting the default behavior based on an absent or empty environment variable means that, when you launch your tests from a local machine, the expected default behavior is to never use an identified tunnel.
Configure your CI server to set the
SAUCE_CONNECT_TUNNEL_IDENTIFIERenvironment variable to
ci_tunnelon your CI host and, if present, any slaves.
You Must Configure the Slaves
CI servers like Jenkins will not auto-configure slaves from settings in the master, so you must make sure that you configure each slave to match the master.
- Set Sauce Connect to start from the CI server with the
tunnel-identifieroption set to
- Set up your test script to check for the presence of the environment variable when it runs.
And that's it. In your CI environment, your code will execute using the tunnel for that environment, or any other that also has tunnel identifiers set. On your local machine, your tests will always default to the dedicated local tunnel
A Default Or Another Value Or Nothing
Suppose that in addition to your named CI server tunnel, you have another named tunnel,
dev_tunnel, that is constantly running that is exclusively for the use of your development team to run their local tests. In this case, the default behavior you want is for your test to check for the presence of a tunnel identifier in the environment where it's running, and if it doesn't find one, to use
dev_tunnel. Because you're checking the value of the environment variable, and not just whether it's present, your test will know when it's been passed an empty value, and, if it has, to overwrite that with the dev_tunnel identifier. This code is actually less complex than the previous example.