While working on some functional tests for a hosting provider, I kept running into an issue where the login test was failing due to a 500 error. It appeared as if the site hadn’t been fully provisioned by the time my test was trying to login.
Initially, I attempted adding timeouts to give the installation process more time, but that seemed prone to error as well since the delay was variable. Also, with a timeout, I would’ve had to make the timeout be the longest expected time, and waiting a minute or so in a test suite didn’t seem like a good idea.
Getting it done
You think it’d be a quick fix, right? If this errors, do it again.
Within minutes, I had found a setting in Mocha that allowed retrying a test. So, I happily plugged that in, ran the test suite again, and it failed…
The issue? The JS bindings for Selenium Webdriver work off of promises, so they don’t quite mesh with the built-in test retry logic. And not having dug in to promises much yet, it definitely took me a bit to wrap my head around a solution.
That being said, there are plenty of articles out there that talk about retries with JavaScript promises, which helped bring me up to speed. But, I didn’t find any that were for specifically retrying promises with Selenium Webdriver in a Mocha test suite.
So, I learned from a couple of examples, and came up with a solution that’d work in my Selenium Webdriver Mocha tests.
The Code
You can find a repo with the code and dependencies here, but for convenience, I’m also copying the relevant snippets below:
The retry logic
This function below recursively calls itself, fetching a promise with the test assertions, and decrementing the number of tries each time.
Each time the function is called, a new promise is created. In that promise, we use catch so that we can hook into the errors and decide whether to retry the test or throw the error.
Note: The syntax looks a bit cleaner in ES6 syntax, but I didn’t want to set that up.
[javascript]
var handleRetries = function ( browser, fetchPromise, numRetries ) {
numRetries = ‘undefined’ === typeof numRetries
? 1
: numRetries;
return fetchPromise().catch( function( err ) {
if ( numRetries > 0 ) {
return handleRetries( browser, fetchPromise, numRetries – 1 );
}
throw err;
} );
};
[/javascript]
The test
The original test, without retries, looked something like this:
[javascript]
test.describe( ‘Can fetch URL’, function() {
test.it( ‘page contains something’, function() {
var selector = webdriver.By.name( ‘ebinnion’ ),
i = 1;
browser.get( ‘https://google.com’ );
return browser.findElement( selector );
} );
} );
[/javascript]
After integrating with the retry logic, it now looks like this:
[javascript]
test.describe( ‘Can fetch URL’, function() {
test.it( ‘page contains something’, function() {
var selector = webdriver.By.name( ‘ebinnion’ ),
i = 1;
return handleRetries( browser, function() {
console.log( ‘Trying: ‘ + i++ );
browser.get( ‘https://google.com’ );
return browser.findElement( selector );
}, 3 );
} );
} );
[/javascript]
Note that the only thing we did different in the test was put the Selenium Webdriver calls (which return a promise) inside a callback that gets called from handleRetries. Putting the calls inside this callback allows us to get a new promise each time we retry.
Comments?
Feel free to leave a comment if you have input or questions. Admittedly, I may not be too much help if it’s a very technical testing question, but I can try.
I’m also glad to accept critical feedback if there’s a better approach. Particular an approach that doesn’t require an external module, although I’m glad to hear of those as well.
