QA Automation at Global Personals

By: Jiazhen Xie

Tags:

  • qa
  • test automation
  • selenium
  • capybara

A summary of test automation achievements at Global Personals. In this post I’ll describe the common issues in our QA automation suite and how we handle them. I’ll also describe how we used the parallel gem to speed up our automation suite.


Test Automation

I joined Global Personals as a Developer in Test in September 2013. Global’s successful automated testing programme had been running for a year before I joined. During the last eleven months, we’ve made quite a lot of further improvements to our framework as well as massively increasing our test library. So here I would like to review what we have achieved and what I have learnt about automation.

Framework

We use Ruby to develop the automation scripts, just like many of our applications. Therefore we use RSpec as the test framework, plus Capybara and Selenium for all the web application tests.

Page Object

Previously to reduce the duplicated code, we had created common helpers. However, adding more helpers caused problems after a few months and we had no idea where to find or add our helpers. After some research and discussion we decided to use The Page Object pattern to build our own DSL (domain specific language). The idea is to encapsulate the page’s behaviours into the page object, which is good for maintaining automation scripts and increasing code readability. Below is an example of the login page object

class LoginPage
  def login_as(member)
    fill_in :memberid, with: member.email
    fill_in :password, with: member.password
    find("button[type='submit']").click
  end
end

Calling login_page.login_as(member) in the automation scripts can log the member in and is easy and readable. It gives a clear code structure where the helpers can be found and added. Another benefit is we can now guess the methods in automation. If you would like to perform a payment with automation, you can start by searching for an existing helper, e.g. payment_page.

Continuous Integration

As part of BDD (Behaviour Driven Development), the end-to-end automation scripts are able to guarantee the correct behaviour of our applications, no matter what changes are made in the application. With our success in writing automation scripts, naturally we moved on to CI. By building our CI system with Jenkins, we can successfully run automation scripts against different environments and branches. The CI results are saved as HTML on the server and can be posted to our team communication application, Slack.

But that’s not enough yet. As good testers we’d like to provide the steps to replicate the issue and the clear result of the failure. So we extended our automation script to take screenshots when an example was failing, and attach them to the HTML. People without any knowledge of the QA Automation application can read the CI results and no longer have to imagine what they look like.

image

So far we’ve managed to run 100+ end-to-end tests within an hour in our CI system (it’s all running in one virtual machine, amazing!), and it’s proven that CI is able to capture bugs in our day-to-day work.

Stability

Since most of our automation scripts are end-to-end tests and our product has a lot of user interfaces, there are lots of web UI tests in the test suite. As a result, we experience many intermittent issues like request timeouts, page not completely loaded, server down etc. These reduce the confidence of our automation tests. Therefore, we need some techniques to help the driver to deal with them in order to increase the stability of our tests.

To handle a timeout or element not found error, waiting is a good solution. There are two types of wait, implicit wait and explicit wait.

Implicit Wait

Simply using Capybara.default_wait_time will change the capybara implicit wait time. Capybara is smart enough to retry finding the link for a brief period of time before giving up and throwing an error. However, changing the global default wait time means automation is slowed down. Thankfully Capybara allows us to temporarily change the time by using Capybara.using_wait_time(5) {}.

Explicit Wait

As the name suggests, we tell Capybara to sleep until the element is there. The code is written similarly to sleep 1 until page.has_css?('#targetElementID'). This is better than implicit wait as we don’t have to wait up to the default wait time when we expect not to have the content on the page. But we also need to be careful with this wait. As if the server is down, the automation will continue in an infinite loop, so we wrap the sleep in a Timeout::timeout(5) {} to the loop.

Speed Up

Last but not least, the speed of automation counts. In our agile process, we add more end-to-end tests to our CI as we add new features. But we don’t want to slow down the CI as in theory, a CI run should finish in an hour as it’s for agile testing. We don’t have an unlimited budget for thousands of virtual machines. Therefore we need some tools and techniques to speed it up.

The first idea is to use a headless driver. Selenium is good while developing automation because you can watch it. But when the automation is running on the CI server no one is watching! We chose poltergeist as the headless driver, it doesn’t load the entire browser and can simulate the user activities.

Running tests in parallel is another technique to speed automation up. We are using parallel gem in our automation suite, it increases the overall automation speed by 20%. And in the best case scenario, one of our automation scripts that used to take 912 seconds, only takes 341 seconds with parallel.

The automation run without parallel:

image

The automation run with parallel:

image

Repeat Tests

Because we are developing with selenium driver, it’s possible that some cases will not work with the poltergeist driver. In order to get the balance of speed and accuracy, we built a repeat mechanism into our automation script. The automation scripts are executed concurrently with poltergeist driver, the failing examples are recorded in rspec.failures file (if any). Then the script re-runs the failing examples using Selenium. This has reduced the number of failures caused by using a headless driver on CI.

Summary

Here at Global Personals we’ve made great progress in test automation, it speeds up regression testing without reducing the test confidence level, really helping software development in an agile environment. Of course there is still lots of work that can be done in automation, for example, catch the JavaScript errors on the page then send the whole error stack to the developers. We will never stop improving the automation because it’s really enjoyable and helpful!

If you would like to work with us at Global Personals, check out our jobs page, we’re hiring.


About the Author

Jiazhen Xie

Joe is a Software Engineer at Venntro. He is passionate about Ruby and QA automation. He's regularly on the lookout for ways to improve his skills.