Empower Quality Assurance by Shift-Left Automation Testing strategy

Shift Left Testing Definition in Miro.com

In miro.com, which is a online world wide lead whiteboard online provider, we have unlimited E2E level test cases due to our customers can do anything they want, such as “create a sticker”, “rotate it” and “add text” etc, we have unlimited combinations of test cases from user’s perspective, it’s like testing a game, the player could achieve anything based on some rules.

In Agile Testing, it’s very hard to split Test Stage isolatedly when we have hundreds builds everyday, especially in our cases we recommend our engineering team to shift whole Test Pyramid to left which not only ensure the quality from Unit test level, but even from E2E UI test level.

Let’s compare our Shift-Left strategy in Miro.com and traditional sprint work

Traditional sprint release

Shift-Left based release

Let’s try to list some differences:

Traditional Sprint VS Shift-Left Sprint

Summary of Whole Test Pyramid Shift-Left strategy:

1. All kinds of testing for new feature are implemented before new code merge into master
2. The responsibility of Quality Assurance is not QA work only, but everyone in engineering could involve in and responsible on it
3. Each feature can be released individually due to all kinds of tests are prepared and executed for each pull request
4. We find regression bugs in early stage instead of wait until the end of Sprint, or waste time on figuring out which PR bring the bug.
5. The whole process is scaleable based on the scale of the team size.

Prerequisite of Shift-Left Testing

Shift-Left testing(especially whole Test Pyramid Shift-Left) cannot be easily switched from the methodology of traditional sprint work. The key is developers, developers can add unit test generally, but how about contract/integration testing and E2E Test(UI/API)? What tool can be convenient for Developers and QA to use together? How can we measure the contribution in different test levels? What would be our Key performance indicator to tell everyone we are in right direction? How can we always use same standards to write tests?

To resolve all of these questions, we analysis from different perspectives:

1. The tool
2. The standards and alignments of using the tool
3. The mechanism to monitoring the tool usage, identify and adjust the direction

Collect requirements about Tool (E2E UI test)

We have different level of testing, but in this article we start with E2E UI test which is the most top level in Test Pyramid. We are asking ourselves what would be the goal at middle of 2021, we are using following steps:

1. Analysing our current E2E UI Test framework which is Java + Selenium, which is a centralised project that everyone who wanna add new E2E UI tests should come to this project
2. Collect honest feedback and painful points from developers about current test framework by group meetings, survey and direct talk.
3. Requirements and Tasks generation based on the feedback we collected and the technical stack of miro.com
4. Comparing the tools together with Developers and choose one, we compared Cypress.io, webdriver.io and Playwright.

Due to the functionality of Miro.com provided is different with other traditional website, we comparing our tools from following perspectives:

1. Coding language: the FE developers are main contributors, so it’s better to choose a tool which uses same language as our Front-end code, our Front-end is in Typescript
2. Browser actions: The most important browser actions from our customers on whiteboard are: Click, Type, Drag-And-Drop, Zoom-in/Zoom-out
3. Cross-domains: Miro works with a lot of different other tools such as zoom.com/google.com, our test should support different domains
4. Cross-browser: Most of Miro customers are coming from Safari, Chrome and Firefox, we need a tool which can test against all of these 3.
5. iFrame support: Miro customers can also create their own plugin within whiteboard, each plugin is a iFrame
6. The speed of test execution: based on Shift-Left strategy, we will run all kinds of tests(include E2E UI test) for each Pull request, it requires our test should be acted efficiently
7. The scalability of parallelisation of test execution: due to it requires fast execution, we need to think about cross-machine parallelisation which can be scale up when we get more and more E2E UI tests.
8. Documentation and Maturity: the tool we choose should well-documented and which could be already recognised by market.

Within this process that we work together with Developers, finally we decide to use “Playwright”.

The story in Miro about Shift-Left strategy(E2E UI)

In miro.com we have hundreds builds every day and quite frequent releases everyday, we don’t have chance/time to test what we did at yesterday, we are using strict 100% test passed rule to allow a Pull request can be merged and a master build can be released which could ensure less bugs be published to customers and ensure the test cases are always be updated and stable.

However, the dream is always far from the fact at beginning. We definitely have a centralised test project that everyone could contribute to, but finally we end with:

1. Duplicated code in the repo and less contribution from developers, few people know the rule and standards of writing a new tests, most of contributors are only QA, Front-end Dev cannot participate in due to we are not very convenient to switch context between Java-based test project and Typescript-based Frontend code
2. Flaky tests around everywhere, due to less standards and rules, the quality of the test cases we added is not enough, finally we end up with rerunning the failed tests to make them pass, it’s not a way to test our product, but it becomes Developers need to test our automation test are passed…
3. Unstable communication between test case and Remote Selenium Servers, browser created by Selenoid can be died randomly, to make selenoid work in a very stable way become expensive
4. The new automation tests are added sometimes after the feature released 1 month after, the value of testing become less and less.
5. Due to less-well structured test report, a failed test report cannot explain what exactly happen, to analysis a failed test case sometimes take up to 1 day, it become even worse when the bug been merged into master branch.

Comparing the Automation testing in miro.com with other companies, we may have several different facts:

1. The growth of engineering department is extremely fast, the amount of PR and master builds which need to be tested can be double just in 3 months
2. We are targeting at frequent release, everyday we have several release, we don’t have time to test what we have released yesterday.
3. The bug need to be caught before PR merge due to we flood of PRs everyday and frequent releases, it would be nightmare when bug caught in master release due to we need extra cost to analysis the error or rollback changes if necessary, even worse, it will blocks a lot of other releases while we are fixing it/rollback it.

We decided to apply Shift-Left Testing strategy

Instead of unlimited scaling up QA team to fit the growth of engineering department, we thought it’s better to include Developers as part of Quality Assurance Lifecycle. Based on our facts, we have our initiative to:

Allow every engineer can contribute and maintain E2E UI automation tests autonomically and efficiently, and catch bug before merge new code into master.

In previous sector, we decided to use “Playwright”, however we cannot directly use Playwright due to:

1. Playwright is a new tool, the interface can have break changes in the future, to refactoring all usage of Playwright in the future will very expensive
2. Playwright is just a UI test tool, but our target is to delivery a tool together with our E2E UI Testing standards together with tool
3. To test collaboration(2 users are using a shared board) we need to adjust a bit of Playwright interfaces to make it to fit our special requirements
4. Playwright can be used in different ways, due to our project will be co-work together with Developers, we need to setup a clear boundary that which part can be contributed by developers, and which part should be managed by QA
5. Playwright just an automation tool, which doesn’t provide a way about test cases management, test report history(Failed test history etc), and no way to monitor the flakiness.

We decided to create our own Lib on top of Playwright

This is a hard decision because we have to add and maintain some extra work on exist framework, but it’s necessary for us to resolve all problems we discussed above.

Our customised tool — Spectator (with Miro.com)

Some short introduction for each component:

1. Bot API is a new wrapper of Playwright API, it designed from “User Perspective”, which supports browser actions and validations such as “bot.wait.untilElementIsVisible” etc
2. Actor API is a wrapper of API calls, to prepare test data, we need to call several Backend APIs to achieve it, such as create an account
3. Page Locators is a layer where to write element locators grouped by page, which is inspired by Page Object Models, but Page Locators only contains element locators, and categorised by “Component”, “FloatPage”, “Page”, “Frame” and “Widget”(which is designed for Canvas widget in our whiteboard)
4. User Steps is a layer to group business user actions, such as “createASticker”, which is inspired by BDD(Cucumber) and Page Object models, it merges the BDD format but put page locator related actions into different layers. We may have a different article to explain in a more detailed way.

So finally, we have clear boundaries between Core Framework contributions and Test contributions, developers will add test with “Actor”, “Bot”, “Page-Locators” and “User Steps”, and some centralised QA will maintain the function of “Bot” and “Actor”.

Finally in this way we have our own customised E2E UI test lib powered by Playwright. We release our tool everyday to developers and QA.

We decide to put test besides the code

If we still recalled, our customers are developers, we want to add E2E UI tests together with developer’s code in same Pull Request. So finally the tests just stay besides the code, developers can easily update them and check them.

The Pros and Cons of 100% test pass strict pass rule

In miro.com we are carrying with 100% pass strict rule to allow a PR merged into master and a publish a release into Production.

This rule looks great but it bring us a lot of problems due to we have a lot of flakiness in our tests. If you recalled we discussed before, developers lost confidence about testing but just to rerun failed test again and again.

But from another side, if we don’t keep 100% pass rule, to figure out a bug after merged into master will even worse due to we have hundreds builds every day.

However it’s a problem before Shift-Left E2E UI testing, because our centralised E2E UI test project is managed outside of developer’s code, the process of maintaining test cases is split with the process of adding new code, as the result, to fix a flaky/broken test become optional.

After Shift-Left strategy applied, due to the test and code are in same repo, and in the meanwhile we execute all E2E UI tests for each PR, so to update test or fix flakiness is mandatory before the PR be merged. In this way, the E2E UI automation tests are always in good-maintained situation without extra planned effort in most of cases.

Last step: To monitor the flakiness

It’s not a small group work anymore, from several QA sit together and decide what we should do, which become work together with hundreds engineers, the communication would become hard. Previously we can have a meeting together with several QA to analysis it, but now we need to provide an autonomy solution for all engineers and ensure we are working for correct direction.

In our cases, we have following ways:

1. ESLINT : to ensure the test are wrote in our expected way. (such as we disallow to use assertion out of test body)
2. AllureReport with history and enough logs and videos: to ensure everyone can understand and analysis why the test is failed
3. “Telescope” one of our own tool to collect test history from Allure report and identify flaky test and share to developers: to ensure we don’t continuously add flakiness

An example test report

The last but not least

In miro.com, our Shift-Left testing strategy is still ongoing, we may have more problems in the future, but I believe we are tracked properly in the process and we are also waiting for YOU to join us and share your opinions and ideas!

In this article, I may made some mistakes, you are very welcome to reply on this article to point it out.