Manually authored tests come in all shapes and sizes, leaving coverage to chance.
Model-generated tests give you full functional coverage based on a proven coverage algorithm.
Are you constantly running over your testing budget and schedule? Do you find yourself feeling like you don’t have the resources you need to fully complete your testing? You’ve probably had the thought: “We just have too many tests!”. You are probably right.
When it comes to managing the testing phases of a development project, there is a fairly short list of things that impact the test count. I’d sum them up as:
- Test Resources People, budget, equipment, environments, etc.
- Test Schedule The period of time available to test
- Test Scope Breadth What will be tested
- Test Length How much is tested in each test
- Test Scope Depth How deeply to test each part of the system
And in most cases, things 1 and 2 are used to determine things 3-5. While there is certainly something to be said for using this approach for pragmatic reasons, the number of tests remains arbitrary relative to the size and complexity of the system under test. This creates serious risk of defects escaping into production.
There is a solution available to avoid this risk: Build a model of your system; specifically, a special kind of model called a Cause Effect Model (CEM). Building a CEM allows you to decide on items 3-5, then automatically generate your tests from the model. This method directly relates the number of tests to the number of requirements and the size and complexity of the system. You can then use these tests in combination with items 1 and 2 to decide which to run in your current phase. So what does this look like in practice?
In a recent experiment, we tasked an experienced Tester to analyze an example user story describing an online credit card checkout function, then write the tests he would run to verify the acceptance criteria of the user story. We then took the same user story and created a Cause Effect Model of the acceptance criteria. The traditional tester authored 25 tests, resulting in only 43% functional coverage. The CEM generated just 12 tests and came in at 100% function coverage. This means over twice the functional coverage with fewer than half the tests. We use the number of covered functional variation divided by the number of total functional variations to calculate test coverage. Click here to find out how we performed this functional coverage analysis or if you’d like to request a coverage comparison.
In summary then, even with experienced testers at the helm manual testing is not very good at determining the number of tests you need nor is it good at achieving high coverage levels. We have a way that excels at both. So if you think you have too many tests, it’s probably because you do. By creating a CEM of the system under test, you can be assured that you have an optimized number of tests with the maximum possible functional coverage.
Make sure your next project is on time and on budget.