Why I no longer use “Cucumber” in e2e automation testing

One of important principal for an engineer, is being to admit our fault. I have to admit that I’m wrong to use Cucumber :)

Before you say “No” to me, to understand the reason why I said that seems like to be the unique way to settle the different opinions we have.

If you read my other article, you gonna find I’m always trying to find different ways to target on quick test, short agile process in a meaningful time-cost and budget-cost methodology to help team delivery a quality product.

The definition of BDD from wikipedia.

In software engineering, behavior-driven development (BDD) is an Agile software development process that encourages collaboration among developers, QA and non-technical or business participants in a software project

What is Cucumber

Cucumber is a tool to support BDD test format, and provide a way to write test case in Given-When-Then statements.

Ideally the test cases are designed among developers, QA and business participants and other non-technical people in Given-When-Then format, and be executed when development is done, so that to ensure the code fits to the business requirements.

All that is being said, but did you notice there is a test case transformation in this process?

The designed test case(left) can NOT be used in Cucumber code(right), it was transformed to a kind of technical code in a Given-When-Then format. You maybe will strongly against me, you would think why we don’t keep same code? Look at below scenario

Feature: Search
Scenario: I can search successfully
Given I am in home page
 When I search "abc"
 Then I see "abc" is exist
Scenario: I can search by auto suggestion
Given I am in home page
 When I search "abc" by auto suggestion
 Then I see "abc" is exist
Scenario: I can search by auto by press enter
Given I am in home page
 When I search "abc" by press enter
 Then I see "abc" is exist
-----------------------------
Scenario: I can search successfully
Given I am in home page
 When I type "abc" in search filed
  And I click search button
 Then I see "abc" is exist
Scenario: I can search by auto suggestion
Given I am in home page
 When I type "abc" in search filed
  And I select search suggestion
 Then I see "abc" is exist
Scenario: I can search by press enter
Given I am in home page
 When I type "abc" in search filed
  And I press enter
 Then I see "abc" is exist

When you search, there are 3 ways to search like above. Comparing first 3 scenarios and last 3 scenarios, the difference is the one before is in business language, and the one after is in action-level language.

Now you have 2 choices:

1. implement every business steps like step defintion “I search” in first 3 scenarios.
2. all in action-level language steps like last 3 scenarios

By using 1st solution, you gonna have step definition explosion after half year, it then becomes very expensive to:

• maintain all step definitions and figure out duplicated step definitions
• ensure QA, developers, business and non-technical participants to always reuse the exist step definition
• on-boarding new team players

By using 2nd solution, it then becomes very expensive to :

• ensure QA, developers, business and non-technical participants to aware all element names used in click/press/etc and their changes in the future.
• to maintain this format of test as Acceptance criterial in business which is hard to read and understand

If you don’t want to use neither 1st solution nor 2nd solution. Then we have to transform the Business language BDD test to Technical BDD test.

So people who designed the test case will write Business language BDD, people who write automation test convert business BDD to proper action-level steps and write technical BDD .feature files.

What’s the values! Let’s see where we waste our time:

1. You need to maintain an extra layout in code — step definitions
2. You need to convert the business BDD and write your own technical format BDD and however no one read it and use it.
3. You need to link your own Technical BDD tests to a well-wrote Business BDD test case in a Test Case Management System.

To do all above is just to make you looks more busy? People may still said:

We use BDD format test as bridge between business and technical

If you are the one of this people, it worth to continue read rest of part.

I , maybe not only me, found that we indeed messed up(or misunderstand from beginning) two things:

• design test case in BDD format
• Implement automation test case in BDD format

Design test case in BDD format doesn’t mean our test code should be 100% matched to Given-When-Then, and using Given-When-Then format doesn’t mean you are in BDD.

Look at following code for same test case as above:

describe("Search", () => {
  it("get result by click search button", () => {
      pageHome().visit()
                .type("[data-testid='search-input']", "abc")
                .click("[data-testid='search-btn']")
      searchResult.should('be.visible')
  });
  
  it("get result by auto suggestion", () => {
     pageHome().visit()
               .type("[data-testid='search-input']", "abc")
               .click("[data-testid='search-suggestion-item']")
     searchResult.should('be.visible')
  });
});

What you get:

• Don’t need to maintain step definition layout, code less and code fast, also test fast and feedback fast
• QA and Dev can both contribute this test
• QA and Dev both can maintain the changes when locator has been changed
• Less maintenance and cheaper, in unit test way, no risk when new team player join
• All team players care about quality is not just words any more.

Then what’s my idea:

• Design test cases in BDD way writing in Business Language in Test Case Management System(Like TestRail, JIRA Test case, etc), So you still benefit BDD methodology, and keep bridge between Technical and Business.
• Implement automation test in Unit test style based on BDD test cases, no more .feature or given/when/then.
• BDD is not just using Given-When-Then in your story! BDD is a process to encourages collaboration among developers, QA and non-technical or business participants in a software project, and provide a standard way to describe business requirements. BDD != Write test case in Cucumber.

If you don’t agree with me, don’t be hesitated to pilot your ideas! For all being said, there is one exception: if your business participants will write code of testing, then let’s use Cucumber and maintain them together from Code-Level with your business guys.