A brief discussion on automated testing under microservices architecture

Several years ago, End-to-End testing by simulating customer behavior attracted a lot of attention from engineers in testing, since 2015, Just Say No To More End-To-End Tests changed the situation, besides, In 2022, a famous video Don’t do E2E Testing made a great reminder again, which highlighted the expensive to execute, high-maintenance, complicated debug process, and longer feedback loop.

In recent years, the rise of microservices architecture has exacerbated the problems of end-to-end testing, when and who to write an End-to-End test, how to maintain it across thousands of microservices, and how to make them reliable, where to run these E2E tests while hundreds of Microservice deployment every day.

In this article, I will briefly discuss the 3 different representative solutions I learned from different companies.

Uber’s way — Shift E2E testing left to the “Integration” level through Traffic Routing management

In Uber’s public post on August 2024, “Shift E2E Testing left”, they provided a solution to test every code and configuration change to core backend systems, called “BITS(Backend Integration Testing Strategy)”. The main strategy is to enable on-demand deployment and routing to test sandboxes before deploying to Production, and individual commits can be tested in parallel.

Furthermore, the “Shift-left” test approach landed on the Production Services, based on Uber’s blog, the Production Service can accept Test Traffic and Production Traffic, it’s close to a solution from Netflix — “Test in Production, the Netflix way”, the solution in Netflix is called “ChAP”.

Both “BITS” from Uber and “ChAP” from Netflix are trying to shift the true E2E test to a controlled, scalable, scoped “integration” test — integrate test multiple services’ functionality and configuration in Production under a controlled scope.

Figure 1: Uber’s solution VS Netflix Solution

By comparing the solution, briefly, the core concepts are:

• traffic routing and traffic management between Production and Testing
• Storage Isolation & Sharing

Dave Farley’s suggestion — Contract Testing

Instead of only saying No to more End-to-End level testing, Dave in this video “Don’t do E2E Testing” suggests using Contract Testing.

The primary strategy is to move testing to the integration level for each service. This involves testing the core functionality of the service using contracts from downstream providers, and supplying contracts to upstream consumers to enable consumer’s contract tests. The approach avoids reliance on real environments and integrates into existing CI/CD pipelines, where most engineering teams already perform unit testing and traditional integration testing.

In 2019, Atlassian proposed one of the implementations for contract testing “Spec First API Development ”, which is a variation of classic contract testing supported by Pact.io, Atlassian combined the contract testing and API design, converting the API design as the source of the contract.

In recent years, Pact.io has converted Atlassian’s approach into a formal way of contract — Bi-directional contract testing and is well-known in the Quality Assurance market.

Figure 2: Dave’s Proposal & Atlassian’s Bi-directional contract testing

Sealights as a representative — Select the right test based on automated Test Impact Analysis

A few years ago, I had the opportunity to participate in a call discussing how Sealights could address challenges with End-to-End testing, particularly focusing on reducing the number of tests while preventing bug leakage.

The main strategy relies on an automated mechanism called Test Impact Analysis in Sealights, which automatically analyzes the impact of each test on the application. This helps ensure that only critical tests and those related to code changes are executed.

If You’re Asking for My Opinion?

These three representative solutions have already been validated by many major market players. Naturally, they come with both advantages and disadvantages.

I want to reiterate some of the key points I learned:

Uber: Testing is a people problem as much as it is a technical problem

Google: Building the Right Feedback Loop, Think Smaller, Not Larger

Google: Managers and decision-makers like End-to-End testing because tests that simulate real user scenarios can help them easily determine how a failing test would impact the user.

Takeaway #1: Quality Assurance serves as an embedded step within Development, so with our developers at the center, I believe it is particularly important to choose an alternative to partial end-to-end tests that is simpler, easier to maintain, and more scalable.

Takeaway #2: As Google said, I also think that we cannot replace end-to-end testing with another type of testing completely. So I would recommend finding a suitable test approach to alleviate the problems brought about by end-to-end testing.

Takeaway #3: In my current context, I prefer to use the first or second method to alleviate the problems caused by end-to-end testing in a microservice environment, while the third method can be more commonly used in non-end-to-end testing, assuming that we have a large number of integration tests.

Takeaway #4: To strike a balance between speed and quality, we can significantly reduce the number of existing End-to-End tests from thousands to just a few by using a “Smoke test suite.” However, this approach may lead to a higher risk of defect leakage. Finding an alternative solution for part of the E2E testing is more crucial than hastily cutting down the amount of executed tests.

Takeaway #5: I also don’t recommend mixing multiple above methods, because each method, in addition to its complexity, will confuse developers, and more and possibly overlapped tests will make automated testing itself more boring and redundant.