Creating an Effective Software Testing Process
An all too frequent experience for IT managers is to have a software release go into production after multiple iterations of testing, only to immediately experience significant defects that went undetected during testing. Fingers point and excuses are made, but the bottom line is that business costs are incurred because the test process did not do its job.
What is test effectiveness?
We define the ability of a QA organization’s test process to reliably detect software defects prior to production release as its “test effectiveness.” The key metric for test effectiveness is simple: number of defect escapes. That is, how many defects escape detection in testing, only to show up in production. If you have strong test and QA processes, the number of defect escapes should be small (3-4) and minor in nature.
Test effectiveness is determined by your testing process. You have one don’t you? In many organizations, the test process is simply whatever each individual test engineer wants it to be. This guarantees inconsistent results and an overall lowering of test effectiveness. Improving test effectiveness starts with a well-defined, repeatable process which specifically addresses these issues:
- Define the coverage criteria that determine when testing is complete.
- Define the process for designing test cases that meet these coverage criteria.
- Define how the test team gets a reliable and complete description of the expected system behavior from which to test.
- Track that the test cases are executed as designed.
- Ensure that defects are tracked and then retested.
How do you manage for effectiveness?
Issues 1 and 2 are by far the most critical to high test effectiveness, yet these are the most neglected issues in most test organizations. Without clear, quantitative coverage criteria there is no way to know when testing is complete. Application of rigorous, measurable coverage criteria routinely results in software with few, if any, defect escapes. Moreover, good coverage criteria can be supported with tools and technology to design and implement the necessary test cases. The test process becomes more consistent and cheaper.
What is the test effectiveness of your QA organization? Many defect escapes or few? Consistent or inconsistent? If test effectiveness is low or inconsistent, examine your test process with special emphasis on whether it delivers test cases which fulfill measurable test coverage requirements. Testing should be an engineering effort, not a creative art.
If you have questions or want to learn more about effective testing processes and technology, contact Critical Logic at moreinfo@critical-logic.com.
Learn More and Stay in Touch:
- To learn about Critical Logic, independent validation and verification, and the Logic Based Development Process, click here
- We welcome your questions and comments. Contact us at: http://www.critical-logic.com
|
|
|
|
|
|
