Automation, an interesting way to achieve the best possible coverage for our test activities

During all my experience in testing world, in several informal chats maintained with different managers and people involved in QA, one of the most recurrent topics of discussion could be this one: how to achieve the biggest level of test coverage in (heavy) systems and applications?

In all these discussions a certain amount of unrealistic (if you’re not working for the genie of the lamp) solutions are offered, that basically can be grouped in two basic proposals: let’s either add as many people as need to our existing test team or  let’s increase our timeframe to perform a test execution cycle until all our tests are completed. As you probably have noticed, those options are not feasible in 99% of test projects, so if you’re looking for a way to provide better insight into on both the quality and risk for a system you should look for other choices.

There are some key elements included on every well performed test process (test planning, assessing test basis, design of test cases or maintaining project testware, among others) that lead to achieve the appropriate level coverage with the resources given. However, none of these elements, under certain circumstances (do no forget this!), help to increase the coverage of our test activities as automation does.

Test automation is that process that could allow us to run big suites of test cases in a short period of time, and also repeat every execution several times. This makes automation a good choice for testing both new and ‘old’ (regression) functionalities of a system, and also, what take us again to the first part of this post, to improve the coverage of our tests without forgetting time and cost constraints, what make us obtain a better picture of the whole quality of a system.

But, to get to the point, is implementing an automated strategy as easy as it seems? Absolutely not! Let’s enumerate some important points to be taken into account:

• Setting up your automated scripts will not be always a fast solution: in fact, configure the needed scripts and adapt them in a way that meets the test strategy, could be a long and demanding task, to which resources should be provided.
• Automated scripts do not remain stable forever, maintenance work is essential, specially for systems (or parts of the system) that are going through constant changes: scripts should be adapted to hold its validity.
• Automation should not be the only approach to be taken into account when testing. Manual testing should continue being important in the process, as it offers the change for  capturing deep or special cases that scripts may forget.
• And last, but not least, the appropriate resources should be dedicated to automation tasks. This means that people with both technical (on the application and programming language used) and testing skills are needed: ‘pure’ programmers or testers are not the suitable people to work on test automation.

So, should we keep us out from test automation approaches for several systems? Not at all! Put an effort in these four aspects when setting up a test strategy could lead us to wide advantages during the whole test process:

• The start-up of an automation process requires big investments of time, costs and resources without remarkable results for the process. However, this initial effort would be compensated by both the bigger coverage and time  and cost benefits that will be achieved in later stages of the process.
• After that start-up, scripts should be maintained, considering what will be the  effort to be done in this activity. For this reason, the reusability of the code and stability on the part of the system that is automated is essential. If there is a positive response to both preconditions, a test automation project is more likely to be successful.
• Integrate both manual and automated approaches could the best option to test a whole system: they could complement each other perfectly. Automation could help to achieve the needed amount of coverage with a limited amount of resources, in the same way that manual testing reach test situations that a script could not.
• Testers could provide their point of view to the automation process, improving scripts, and ‘finding’ spots that a ‘pure’ developer may not find.

So, taking into account what we have mentioned above, and under certain circumstances,  automation could be that way to achieve coverage that would be impossible with only a manual approach.

Más información:

Isaac Álvarez Diz – Test Lead – Sogeti España

 isaac.alvarez-diz@sogeti.com