Automating Test Execution: What You Need to Know.

March 2007

As CIOs and software development managers seek to lower the cost of testing, one of the first proposals to be put forth is the automation of test execution. That is, automate the test cases so that QA staff does not have to spend hours or days manually executing them. Sounds like a no-brainer, right? With a proper evaluation and implementation, automation can be very effective in lowering testing costs. But ensuring success requires assessment of a number of important factors that may not be obvious from the information that test automation vendors provide.

Test Case Maintenance

Compared to a human, automated test tools are the dumbest testers possible. The slightest difference between what the automated script expects and what the application does will cause the test case to fail. This imposes a much higher maintenance burden to keep test cases completely up to date. The slightest change in a screen text must be reflected in all automated scripts or they will fail. So factor the need for a rigorous maintenance effort into the costs of automating tests.

By the way, do not be fooled by vendor assertions that maintenance and technical expertise can be avoided by simply "recording" a manual execution of the test and then playing it back. Such test scripts are very sensitive to changes in the application or data and are very expensive to maintain.

Technical Expertise

Automated scripts are implemented via programming, usually in a language that is proprietary to the specific tool. Thus, the automation effort requires creating and maintaining a specialized programming staff of sufficient size to handle writing and maintaining your inventory of test scripts. New, keyword-driven automation tools can shift some of the burden for scripting back to test engineers, but you need to evaluate keyword tools carefully to determine the level of technical support required.

Testability

For automated scripting to work, the scripting tool must be able to "see" the objects on the screen or access the data values in a database. Some screen design practices can make this very difficult, especially if you are automating tests for a third-party package and have no control over screen design or object naming conventions. So be sure to confirm how your applications interact with automation tools you are considering.

Taking all of these factors into consideration, automated test execution can still provide significant efficiency gains for the testing environment in your organization. In particular, the new "keyword" oriented test tools can simplify the creation and maintenance of automated test script.

Making the implementation work

Once you have selected an automation tool, take the time and effort to integrate it into the overall test process. Be sure that the following exist:

• A test data setup plan that matches the needs of the automated scripts.
• Test environment properties that are conducive to automated scripts (system dates, database resets, etc.).
• Criteria for when in the test cycle test cases get automated.
• Enough time allocated to review run logs and automated test results.

Also, ensure that there is strong correlation between the tests as they are defined by the test engineer and the test scripts created by the scripting technicians. Left to their own devices, scripting specialists will often "re-author" the tests for technical expediency to the point where they no longer reflect what the tester originally intended.

Finally, remember that automated execution tools do not improve the quality or coverage of the test cases, only the efficiency with which they are executed. If testing is not finding the bugs, then consider changes in test case design processes and tools.

At Critical Logic, we have advanced expertise and technology for all phases of QA and testing. Call us with your questions.

Experience the difference of getting it right from the start. Visit critical-logic.com or email moreinfo@critical-logic.com for more information.

Come check our booth at the upcoming Software Test & Performance Conference
April 17-19, San Mateo, CA. (Booth #5).