Do you start to self-assess test automation maturity?
An idea to kick off automation test can come from different situations in different kinds of teams, but they went to the same end with similar doubt. Did our automation test works? How much does automation test contributes to business? Is it necessary to decrease or increase the investment to automation test for next financial period?
Those question are tricky, but to always get back to these questions in a proper way can lead the automation test to the right direction from business and technical perspectives. So I want to share that how to self-assess test automation maturity continuously based on my experience in this article to help you to get some new ideas, and to help you to expose potential automation problems in early stage.
Count of bugs found by automation?
The amount of bugs is the quick and direct output and contribution to business in short-term of QA works(of course QA is not only responsible for bug catching). Assuming your bugs are manged by Jira, thus we can use Jira Labels to track and count the bugs which found by automation test every sprint.
However there is a drawback by using this measurement. The developers can’t ensure that they bring same amount of bugs every sprint :), some sprints may be quiet buggy, but some sprints can be very peaceful. Thus the amount of bugs found by each sprint cannot tell you that how maturity the automation test is. Maybe you are lucky enough to catch 100 bugs in Sprint 1, but nothing found in Sprint 2.
Catch Ratio: bugs found by automation / All bugs
We have 2 kinds of Test approaches: “Box” testing(White box and Black box) and Exploratory approach, so the bug list is composed by bugs found by manual test(can found by clients, business, any one), and that found by automation test. So we can use contribution ratio as:
contribution ratio(%) = count of bugs found by automated test / all bugs * 100
The automation contribution ratio will not be impacted by the developers performance, and which is an objective assessment.
By using automation contribution ratio, we expect the ratio will be increased smoothly by expanding test coverage sprint by sprint. But it won’t tell you that the automation team sprint work is in a right direction continuously when you only check contribution ratio. For example, we add 200 new test cases in current sprint, but the automation contribution ratio is decreased slightly, it can potentially tell that the automation sprint work doesn’t focus on buggy part based on 80/20 rule.
80/20 rule:
You will find 80% bugs in 20% codes or features.
Automation Test Trend
— Combination of contribution ratio & count of test cases
It combined the automation contribution ratio and the amount of automate test that we have into one chart.
If the automation contribution ratio is increased by the amount of automated test expanding, we can tell that the automation work within the sprint is on the target, otherwise potentially not.
To Sum up
We can self-assess the test automation maturity by 3 ways above based on my experience.
The automated test maturity is not only be used to answer these tricky questions, but also helps the team to keep confidence by showing all work in transparent way. In the meanwhile, everyone in the team has same target, everyone can always correct the directions by measuring the maturity sprint by sprint, to ensure their works are meaningful and value maximization in limited time. Besides, different maturity stage can be seen as indicator as well, which let you know that whether it’s a good time to start big refactoring or introduce new automation test features, that is also an important part of Risk Management to ensure the improvement of automation framework will not downgrade the quality work.
Do you have other ways to self-assess test automation maturity? I’m willing to see yours , and welcome any suggestions about the way mentioned in this article.
You get some ideas after reading my article? Then maybe a 👏?
