Search For Answers and Frequently Asked Questions
Sauce OnDemand
How does Sauce OnDemand work?
With Sauce OnDemand, we leverage Selenium's client/server architecture, adding the internet in the middle between your local scripts (made using Selenium's client libraries) and the Selenium Server (which is in our side, as the browsers are). On your side you will have several test scripts like this, running in your machines, creating the reports you need. Off stage, the script will send us step by step what the browser needs to do and wait for the response to know if everything went ok and to move on to the next browser action.
To get the full picture, see How Sauce OnDemand works
What is the billed time per-test composed of?
For every test you run, the video time (and therefore test time) will be smaller than the final job duration (what is billed by Sauce). This happens because after the test finishes, our test server will remain fully dedicated to that respective account as it is processing video and screenshots and uploading everything to make it available for later access through our website.
Is important to point out this time will only be necessary in our servers and in the meantime the test scripts in your end will have moved on to the next test to run, so there shouldn't be any impact in the overall test suite execution time.
We believe video and screenshots are great features for which we have them on by default, but if that's not the case in your context and you are interested in cutting off those extra seconds, you can always disable such features (video and automatic screenshots) on a per-job basis.
Here's our documentation on how to do that:
Disable video recording in OnDemand
Disable automatic screenshot capture
What OS and browser combinations do you support?
We support Firefox, Internet Explorer, Google Chrome, Safari, and Opera on Windows, Linux and soon, Mac. Check out our complete list of browser and operating system combos for details.
Will my firewall block your service?
No. By using Sauce Connect, you can connect to our service securely without opening any ports in your firewall.
Can I use Sauce OnDemand if my tool is behind basic HTTP Auth?
For HTTP Authentication, we behave as Selenium does, leveraging the use of the RFC 1738 specification for which you can enter that information within the url. So you just have to change the url in your tests to include the Auth info. Here's a example: http://username:password@www.mydomain.com/secretarea/. You can also read more about this here. This should work with all our browsers, but if you find any obstacles with your site, please let us know.
Note: in case you don't yet have any HTTP Auth setup and all you want to do is have Sauce OnDemand testing your internal app, we strongly recommend using Sauce Connect instead.
How do you handle HTTPS and Certificate Exceptions?
Please check our detailed table describing our browser support list for HTTPS and self signed certificates.
Do you support Mac OSX?
We're excited to announce that we're currently working on Mac OSX support! Keep an eye out for an announcement soon on our blog about this :-)
My tests run serially. Can I still use Sauce OnDemand?
Yes. With Sauce OnDemand, you'll benefit from our many browser configurations and features such as live video recording.
Consider the advantages of parallel testing. It's faster and you may find your tests are less fragile and more capable of pinpointing a problem.
How do I run my tests in parallel?
Check out our various blog posts on how to do that with different programming languages: running selenium tests in parallel.
Can I use Sauce OnDemand for load testing my app?
Sauce OnDemand is meant to be used for browser-based functional testing (also sometimes called "acceptance testing" or "UI testing"). And even though parallelization inherently causes load, the features other services meant to be used for load testing provide are probably better for your needs. We're friends with and fans of BrowserMob. They're much more focused on load testing and the operations side of things than we plan to be.
How is Sauce OnDemand different from Selenium Grid?
Check out our feature list, we're pretty proud of it :)
Can I reuse a session within multiple tests?
One of our main criteria for a mature set of tests is "test independence". If your tests are completely independent from each other, the complete test suite already will be prepared for scaling. And believe us, scaling is not far in the future; once you reach a medium number of tests, the time needed to run them serially will exceed the practical limit and:
- Your team will stop running the tests as regularly as needed because it takes too long
- The tests will start eating more and more of the development time
This can also depend on the context. In case it's really needed, just prevent the test framework on your side from calling the start() and stop() commands between tests.
Troubleshooting
Connection issues (my tests seem not to be able to connect to Sauce)
A general problem some of our users face is when outgoing connections from their private network on port 4444 are blocked, causing the tests to not reach Sauce OnDemand. As a fast solution to that problem, Sauce OnDemand also listens at ondemand.saucelabs.com:80. Just use ondemand.saucelabs.com as the Selenium host and 80 as the Selenium port.
The video seems to be missing even though the test finished
Videos missing right after tests finish are caused by the need to first post-process and upload the recorded video for it to be accessible to our users through the web interface. The time these tasks take will depend on the duration of the test (long tests will produce longer and heavier videos). A reasonable estimation is that video processing + uploading is going to be around 30% of total test time. This means that if your test took 1 minute total (between browser start and shutdown), the video will take 20 seconds to be ready for you to watch. In the same way, if a test takes 10 minutes, it may take us up to 3 minutes to have the video ready.
Please let us know if you find videos are taking longer than that to be ready.
Open Commands are timing out, even though I see the app loaded in the video
This is generally caused by a connection gap or a problem with the application's server handling requests incorrectly. As a very first step, you should proceed with a deep analysis of the network traffic. If you make it automated and run several tests at the same time, you will have higher chances of replicating the error.
Another good recommendation is to try out the captureNetworkTraffic command (which requires the Selenium instance to be started with the option captureNetworkTraffic=true and your test to use Firefox). This will let you pull the request info back out as JSON/XML/plain text. Then you can parse that content and find any problems.
My tests are taking too long to start. What should I do?
We're constantly working into making our resource allocation as slick as possible, but in certain times when our service is under very high load, this could take longer than expected. Please check our status page to see if there's an ongoing issue and let us know if you find this happening too often.
Tests failed on my end appear to be passed in Sauce OnDemand!
Because of the client/server architecture Selenium remote control works on, there's no information about assertion results going on the server side (Sauce OnDemand in this case), which means that in a step that your test will be validating for the title of you AUT to be "My Shiny WebApp's Title", all that OnDemand will see is the request to get the title from the current page and it will only return the result, without even knowing what's expected.
Your test:
assertEquals(sel.getTitle(), "My Shiny WebApp's Title");
Sauce OnDemand:
Command requested: getTitle()
Result: Your Page's Title
Notice that we use the same criteria for other kinds of failures, like a Selenium exception for trying to click on an non existing element. The reality is that tests in your end could be coded in a way so that such failure wouldn't end up in a failed job (e.g. try-catch and continue with the test in a different way).
Good news is that it's still possible to let OnDemand know what actually happened with your tests! Check out our new Pass/Fail API to do it from within your Selenium tests.
Your service is down. What should I do?
We're constantly checking our service and we're probably already aware of it. Please check our status page and let us know if you don't see anything reported there.