Seeing more failures than expected since parallelizing your tests? This topic provides a few suggestions of things to look for when debugging your test code.
See the following sections for more information:
Flaky Tests with Random Failures
- Tests “randomly” pass and fail each run, or throw transient exceptions
- The number of unexpected failures is related to the number of concurrent threads running
- When run in a single thread (not parallel), the tests pass or show expected behavior
- Use of static shared members in test classes
- Use of static members in PageObject classes
- General resource contention on remote site due to issues with local test classes
Depending on your specific framework and test structure, you may want to try these steps in a different order.
- Run your tests in a serial fashion and get a baseline
- Repeat the process with increasing parallelism (increasing the number of concurrent threads) until you see consistent failures
- Do a thorough review of static components in your classes and avoid using the static keyword where possible, and use
ThreadLocalclass members when needed
- Look for resource contention and race conditions by printing resource references to standard output along with thread ID’s. In Java you can use this method to get the ID of the current thread:
- Separate test framework from test content by reducing tests to simple actions with randomized data, and check for test thread isolation. For example,run a Selenium test where you enter randomized text input into a field, delay, and read back to confirm. If there’s interference from other tests the test will fail.
Add exception handling code around critical sections that present the errors. Going through your stack trace will greatly help with this. A sample exception handling wrapper would look like this in Java:
Variable Test Performance and Performance-Related Failures
- Test execution times vary drastically from test run to test run
- Unexpected failures are observed, followed by poor test performance
- Performance improves with reduced thread counts
- Tests failures due to timeouts
- Test runner freezes and becomes unresponsive
- Resource issues on test host
- Resource/capacity issues on test target
- Network infrastructure issues limiting the required bandwidth
- Turn on debugging features of your test runner. For example:
. mvn -X <your build command>
- Run tests with increasing parallelism (increasing the number of concurrent threads) and monitor resources like memory, CPU utilization, network utilization, and disk space (if applicable) on the test host. If any of these resources reach critical levels, adjust test volume accordingly or add resources.
- For your thread count, you can start with “1.5 x # number of CPU cores” as a starting point, and experiment to find the number that would work for your setup. A modern computer would have at least 4 cores per CPU.
- Augment memory allocation for your build process, depending on your needs. For example:
mvn -Xms256m -Xmx1024m <your build command>
- Monitor resources like memory, CPU utilization, network utilization, and disk space (if applicable) on the test target to make sure the machine can support the traffic, and scale as needed.
- Do a thorough review of static components in your classes and avoid using the static keyword where possible. These are not garbage collected, and consume significant amounts of memory even when not in use.