Automation in the real world: Part 1

By: Saurabh Sahu

Tags:

  • qa
  • automation

Prologue

In most organisations QA usually is an after thought and automation is always introduced late. But let’s face it, automation usually arises more often out of need than prior planning in small organisations. This view might stir a reaction from the new breed of Dev in test and Agile Testers out there. I can imagine them taking a step back saying “How could you possibly say that?” or “that’s a poor approach”.

Let me paint a picture of a new start-up company embarking on its journey, solving the world’s problem with technology. Ambitions are huge, stakes are high and the pot of money — small. It does it’s best to get a foothold, creating the smallest and most efficient and bullet proof piece of software it can build (or may ever build again in future). Little acorn. At this stage the company might not even have a QA team.

But as the company builds on top of the existing platform, adding layers of functionality to make the product more attractive to the end users, more complexities are added. This is where everyone starts to realise, going forward they will need a QA team. Enter, first QA person. In this first phase, the focus of the business is sustainability and frugalness. This comes with the expectation of QA to increase quality, make release cycles efficient and time to market short. The QA person may strongly advocate investing in automation from the start, but needs to compromise. Usually outnumbered and new to the team; they give in or do some token automation without careful planning or strategy which will have its own issues later on.

With the software’s popularity growing and in turn the team growing, the feature list ramps up quickly. Before you know it, we are talking a lot of buzz words in the QA team — cross browser support, devices support, performance, security, responsive web, test management, release management, versioning, documentation…and of course automation. In the scheme of things automation is already on the back foot owing to the large feature stack and either the team is thinking how to automate? or now find ways to realign its current half-baked framework and existing scripts to continue forward. If this situation of disparity between automation approach or the lack of and an ever increasing stack of complex features continues, the gap starts to widen.

But this is to be expected in a small to medium sized technology company. However, what happens from here is what determines the QA team’s success or will it be an ongoing battle with automation?


Past

At Global Personals the story was no different to any other small to medium sized enterprise. Although now about 200 people strong, Global Personals started as a small tech company trying to find its feet in the big, harsh and very competitive tech world. Humble beginnings - Little acorns. The company saw a rapid growth and our dating platform became very popular, fast.

The QA team soon realised they needed to have automation. The fastest and best method available at the time was applied: to implement automation using RSpec with the help of various gems. The framework also uses Capybara and Selenium. There was also a CI server for on demand script execution setup. The RSpec framework worked well. Tests were well-written considering the vastness and complexity of the platform. More importantly, the QAs took a more important position in the company and the team and its processes matured. Adding to it, the shift to agile placed QA at the heart of the development process and made automation imperative.

An example of our RSpec test

feature "Logging in on desktop", :essential => true do
  WLD::Test::Constants::SITES.each do |site_id|
    context "for site: #{site_id}" do
      before(:each) { platform.change_site_to(site_id) }
      let!(:member)  { platform.get_member }
      after(:each)  { platform.reset_session! }

      context "with a valid email" do
        context "and valid password" do
          before(:each) { platform.login_page.login_as(platform.member) }

          scenario "member is on the site homepage" do
            expect(platform.current_path).to have_content(/\/member\/|\/member\/my\/profile\/wizard\/|\/member\/remainder\/profile.cfm\//)
          end

          context "member tries to navigate back to the sign up page" do
            before(:each) { platform.visit ('/signup.cfm') }

            scenario "member is redirected back to the site homepage" do
              expect(platform.current_path).to have_content(/\/member\/|\/member\/my\/profile\/wizard\/|\/member\/remainder\/profile.cfm\//)
              expect(platform.current_path).not_to have_content('signup.cfm')
            end
          end
        end

Another good thing that happened early enough was the introduction of Page Object modelling. Using the PageObject model had made the test clearer, extendable and easier to keep up with any markup changes. See this post for more details.

So far so good. However, as the number of RSpec examples (i.e. tests) piled up the tests became more complex and embedded in the examples deeper and deeper. See example above. They were hard to read and thus harder to assess coverage and difficult to maintain. The framework did not lend itself to parallelisation and suffered with long execution times due to linearity of the tests. With no review of the current state, the automation tests and the framework grew organically.

If you’re interested in seeing how we’ve begun to overcome some of these challenges, I’ll be writing a follow-up post in the next few days, so please keep an eye on this blog, or on our Twitter feed, for updates.


About the Author

Saurabh Sahu

Saurabh Sahu is the QA Manager for Venntro and heads up the globaldev QA teams in Windsor and London.