Running Selenium test in Paralleled Cypress way
I guess most of QA easily misunderstand 2 categories in an e2e automation test project: test framework and test runner. You may thought I’m talking same thing, but not.
Test Framework: a library to control the application, like webdriver, selenium, cypress.
Test Runner: a tool to execute your test case, like mocha, cucumber-js, jest, CypressRunner, TestNG and etc.
Sometimes you can select a test framework with different runner, such as you can use webdriver with cucumber-js or mocha, but sometimes you cannot, such as Cypress & CypressRunner.
When the topic comes to Parallelization execution, we are mainly talking about How test runner support it.
Cypress Parallelization
The Cypress Dashboard worked as the test planner and scheduler, it will will tell what spec file to be executed in each TestRunner within CI Machine. Finally you will get an integrated test report within Cypress Dashboard.
in a r, like following:
Well, Cypress prepared everything for you, what I need to do is to focus on writing test case, and providing a .gitlab-ci.yml file. However, we need to pay Cypress Dashboard but it’s cheap enough because the test compute resources are coming from your gitlab pipeline runner.
The parallelization is supported by CypressRunner, and you have to write test cases with cypress test framework.
However, cypress doesn’t support test-case level parallel, I believe it will be same reason like mocha, we only can execute spec files in parallel. As the results, some spec file will be ended earlier, but some of them will take long time. (even Cypress Dash provide load-balance mechanism to optimise it)
Selenium w TestNG/Cucumber-js Parallelization
TestNG has multiple level parallelization in JAVA language, and Cucumber-js also support test level parallelization. However Mocha only can run in file level parallelization.
In this way, to run selenium in parallel can be done by following way:
This process is more complicated than Cypress, our aim is to test localhost:3000 within CI pipeline, so we have to do something like “Local Testing proxy” as what BrowserStack did, to let our Cloud Selenium Grid can be accessed from Selenium Grid cloud.
I believe this way requires more effort and risky by network proxy, and if test cases executed in CI pipeline are unstable. then it doesn’t make sense.
All is being said, this way can only be used in Regular Regression tests, but it doesn’t fit for CI pipeline test as the setup process is too heavy.
The organiser/planner(CI machine #0) is part of CI/CD pipeline tasks, which should:
- Test Runner should support to run test by given spec/feature files, like “ — spec a.ts b.ts”
- Test Runner should support dry-run that provides a view about which test cases/test file need to be splited.
The output of organiser(CI machine #0) will be X files with different names(3 files in above example chart) , and be archived within pipeline and be used by next 3 CI machines. Each of those files contains different list of spec/feature/tag that need to care about only for that given machine.
From CI machine #1-#3, all of them do:
- Start App at localhost:3000(or other local domain)
- Start selenium standalone server
- Run test with given file(get the file by name respectively)
All is being said, we can do parallelization in same way as Cypress, but we don’t have dashboard to check integrated test report from CI Machine #1–#3 as we don’t have a CypressDashboard which can receive the data.
However, we have 2 choices:
- Add one more CI/CD pipeline step, to collect the test report from each CI machine, and merge them.
- Create an independent Report Server, that can receive the test report and merge them , that act as same as CypressDashboard.
The last topic then, would be how to make your Selenium test become very stable, that to avoid test case issue happened blocked CI/CD pipeline…
