The way of quality assurance in my daily work

I want to tell you a story about how to lead the good quality assurance in my previous daily work that based on my limited knowledge. It would not be the best way, but it could improve the quality assurance in long term. Especially from an automation guy’s perspective.

This article reviewed my high level working process, and try to explain the business value of QA teams I contributed in past 10 years by explaining the detailed daily work process, and try to explain why I don’t mention Automation testing in whole process at the end of this article .

We start from the initial topic: WHY do we need quality assurance?

Dev cost in each period

In the left chart, we identified that to fix a bug in Maintenance Stage is quiet expensive since Dev team need to stop current work and switch from current topic to a old topic again, in the meanwhile, the issues already become to client-facing issues which did impact client’s usage and business work.

Thus, that’s the original reason that people start the role of QA in the team to resolve this issue: how to catch bug in early stage and decrease the long-term maintenance cost.

We come back to another topic: What is the business value of QA

Product owner plan/define/collect the requirements for the team, Devs make the Product owner’s dream become true, Scrum master define and help the daily working process in an agile way, QA team ensure the quality of the product we are delivering.

In short, all teammates ensure that team can delivery a high-quality product in efficient way to match business requirements.

However, when we come back to the definition of QA business value — “ensure the quality of the product”, it seems that we are facing a quite big topic and there is no way to measure our work: “did we already ensured the quality??”. From QA methodology, we have a lot of types of testing, Black box testing, White box testing, or divided by test purpose like Regression Testing, Smoke testing.. or even we have End-2-End testing, API Testing etc… So assume you did all of them, can you tell your business that “Hey Team, no bugs, let’s release”. As a professional QA, I won’t say that, but I would say:

Hey Team, Based on our current test scope, no blocker bugs, let’s release!

Thus, what we are missing in previous statement — Test Scope. So to match the QA business value, our QA main target is: Define and update test cases based on the acceptance criterial for positive and negative follows(input and scope), catch more bugs as much as possible in early stage(output).

The measurement of QA work would be the percentage of

QA contribution Measurement = (bug find before release / bug find before release +. bug find after release)

QA Effort Measurement = the amount of updated/created test cases

Thus the problem should be solved by Quality assurance is:

How to define the Test Scope properly that can cover most of critical scenarios?

I want to emphasize one statement here: Quality assurance cannot promise to cover 100% bugs, so the measurement of QA work is to understand the team healthy, such as if we said we have 1000 test cases, but none of them catch bugs, so it means there is “problem” about the processing, as a QA lead, we should immediately to solve the problem.

While, we reviewed the root reason why we need QA, and figured out what’s the business value of QA . So at this moment, all our work/effort/budget should focus on how to make QA team meet the business value, and continuously review QA output to adjust the direction back to the right way.

Let’s come to the way that I’m working : How do QA “ensure the quality”

From the chart we know that: QA team participates in UI Tests and Service Tests, DEV responsible for Unit test, and sometimes QA and Dev, even with PO, we together responsible for UI tests and Service Tests. (Performance testing is out of scope in this article, will discuss it in different article)

So generally we can define QA work in 2 types from functionality testing perspective:

1. End-2-End testing (UI Tests)
2. API Testing (Service Tests)

Let’s have a look about how to handle these 2 types of testing from high level.

* Collect acceptance criterial & Define Test Cases

We know our requirements in more detail, we define the test cases better! Test cases describe the SCOPE about what we SHOULD test based on the given acceptance criteria, which contains Positive test follow and Negative test follow. The test cases would be defined during the “Sprint Refinement & Planning meeting”. (It could be just short sentences in a Excel sheet, and enrich later)

Working together with your Scrum Master and Product owner, to find a best way to inject QA process inside is very important. Some key points for this process:

1. Don’t put all QA to this meeting, we only need 1 or 2 person to understand the requirements, and convert the business requirements to Test Cases in a common language and share to rest of team.
2. The test case input should not only come from you, but also from Dev, Product Owner, and everyone should participate in to figure out the test point.
3. Don’t start any test if the ticket doesn’t been reviewed, in some cases when you start guess acceptance criterial, you already start put risk for the future.

* Enrich and review the tests with your QA team mate after

We said that “Everyone should care about the quality”, BUT QA team would be the principal team to responsible for it.

We maybe cannot completed all the test case design with all the detailed steps in one meeting, thus to enrich the designed test cases after the meeting is very important to let rest of team mates to know that, there would be some extra points should care about:

1. Ensure new test case is independent, and doesn’t duplicated with others
2. To verify the new test case can be merged into other existed tests
3. To verify the new test case have enough test steps and test data, and linked with main story.

4. Share all the test cases you designed to rest of QA team, ask for a review

5. Share the designed test cases mapped with story ticket to Product Owner, Dev, and Scrum master

Cucumber test for E2E Test

* Run test cases and report bugs

After we have the scope, the next important part is to execute your test, from business perspective, if we don’t consider the budget/time consuming, we don’t care about wether it’s manual testing or automation testing, we only need to care about following 4 parts:

1. We respect our designed test cases, do it step by step, and match the expectation at the end.
2. After the designed test case execution, we start random exploring testing without any pre-defined steps.
3. Report bug immediately when issue is found, leave more time for Dev and Product owner to decide the priority and implement patch.
4. Generate test report for all sprint testing, let everyone know which bugs are still exist, and will will be released. (Sometimes we accept some risk)

One of the ways to show the link between Story/Test case/Test execution/Bug together in JIRA

* Review test cases and test process among QA team/Cross team at end of each sprint

After execution and bug reports, a very important step is that to come back to review the new designed/updated test cases again for the passed sprint. All the QA mates should aware the changes, and in the meanwhile, we also do QA Team internal Retro, to figure out is there any space to improve the QA process, we need to care about following topic:

1. Ensure everyone understand the changes of the test cases, line-up with everyone per sprint, don’t leave the gaps.
2. Share the idea about current QA process by every mate
3. Collect feedback about QA team from Dev and PO during cross team Retro meeting
4. Restore our knowledge about the test cases and to avoid to create duplicated test cases in the future
5. Review the target of last sprint, and define the new target for up-coming sprint

Test Case Repo (X-Ray in JIRA)

Achievements and Benefits in long-term :

By above way, it ensures that our QA work can be measured from following perspective with our QA Target:

1. Test case coverage, increased/updated every sprint
2. Bug catch rate, increased/decreased
3. Team efficient, release in time with good quality?

Achievements sprint by sprint:

1. We try to don’t miss any test points as much as possible in early stage with all team member’s help
2. Based on the scope, we find more and more bugs in early stage
3. Exclude the test scope, we did exploring testing to find potential bugs in complex test process manually, and to innovate new test scenarios.

To check how we measure automation business contribution, pls move to here: https://medium.com/@xuanzhaopeng/the-way-of-quality-assurance-in-my-daily-work-d40395a48052

In the meanwhile, we get extra benefits for rest of team(except we catch bug earlier…):

1. We use Test cases to describe product’s acceptance criterial which will be updated once existed function been changed. Every new team member(no matter Dev, PO, SM, QA) can understand how our product works by reading test case (Risk Management), it’s not necessary to to ask the acceptance criterial around the world!
2. Dev, QA and PO has same knowledge about the feature by design test during Refinement & Planning meeting, no misunderstanding.
3. Everyone participates in quality assurance, QA work become transparent.

As an Automation testing guy, Why I don’t mention automation in my daily work? :

I was working for QA Automation testing in past 10 years in different countries and sectors for different types of product. I have rich enthusiasm to test everything by automation tools! BUT! First we are QA, then we are automation guy. The automation testing won’t change anything for above process, the only difference is:

Automation guys create automation test eco-system for everyone to write and run test in short time, or even generate test report. And the work can be repeated again and again without extra effort.

That’s it.

It’s like that you wanna eat some food, you can choose folk and knife, or you can choose chopsticks, they are both used for taking the food to your mouth. If I told you I will invite you for great dinner, do you really care about that we will use folk and knife , chopsticks or others?

At the end, if you are still not sure that does your QA team contribute well to the business, then ask yourself:

1. Did your product maintenance cost goes down (no matter from finical part or team effort)?
2. Did you always release product in time based on plan? Do you always has last minute bug just before release?
3. Did most of bugs found in early stage before release?

Hope this bit of my own daily work can help you or your team have new vision about Quality assurance work no matter in automation or manual testing.