UI Test Automation in Adobe Commerce
Automated testing is an important part of any software development cycle. When testing UI components, everyone has experienced unexpected errors after deployment. Sometimes it's affected functionality that wasn't thought through, sometimes it's specific cases in a database, and many other possible reasons.
This is why automated testing is an important part of any software development cycle. When testing UI components, it is highly beneficial to run a smoke test after every deployment, or better yet, after every merge request (MR) to make sure the reason for an issue is present. If you work with Adobe Commerce (Magento), you've probably heard of the Magento Functional Testing Framework (MFTF), which can also be integrated into your CI/CD process.
MFTF provides a large list of tests to check a variety of settings in different configurations and act on the front-end (FE). Unfortunately, it also has a downside: the tests are created for vanilla Magento and it can take some time to adapt them to your project, especially if you use a lot of extensions.
So, in order to be more flexible and have tests that fit our projects exactly, we decided to develop our own UI tests from scratch.
Architecture
Pytest is a Python testing framework that can be used to create all types of tests, from simple to complex and scalable test cases. Pytest can be easily combined with Selenium to implement the Page Object pattern. Page Object is a design pattern that has become popular in test automation to improve test maintenance and reduce code duplication.
A page object is an object-oriented class that serves as an interface to a page in your AUT. Tests then use the methods of this page object class when they need to interact with the user interface of that page. The advantage is that when the user interface of the page is changed, the tests themselves do not need to be changed, only the code within the page object. Then, all the changes to support the new user interface are in one place.
The Page Object design pattern offers the following advantages:
- There is a clean separation between test code and page-related code such as locators (or their use if you are using a UI map) and layout.
- There is a single repository for the services or operations provided by the page, rather than having those services scattered across tests.
The pattern fits perfectly on a website based on Magento as it usually has highly specialized pages with fixed and specified elements, e.g. a product page usually has price, quantity and add to cart button that are not displayed on any other page, on the other hand email/login and password fields are displayed only on the login page.
Nevertheless, all pages have some common elements like header, footer and messages. These common elements can be described in the main Page class and page-specific elements in inherited classes.
Before we describe a page interface, we need to start the web driver and call the page. For this we implement the next constructor and the basic methods that can be used on each page:
from abc import ABC
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
class Page(ABC):
browser: RemoteWebDriver
def __init__(self, browser, host, wait_time):
self.browser = browser
self.host = host
self.url = ""
self.body_class = ""
self.wait_time = wait_time
def load(self):
self.browser.get(self.host + self.url)
def is_current_page(self):
if self.body_class != "":
return self.find_element("body.{}".format(self.body_class))
else:
return True
def find_element(self, selector):
try:
return self.browser.find_element_by_css_selector(selector)
except (NoSuchElementException, ElementNotInteractableException):
return None
A very good tutorial on how to create UI tests with Pytest and Selenium can be found here: 
In this article I will only explain how to apply the approach described in the tutorial to the needs of Magento applications. In short words here "browser" - is an object of WebDriver (we can support different WebDriver and run our tests on different browsers each time, which are set up in the json file).
We also implement the most commonly used method that checks if an element is displayed on the current page:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
def is_displayed(element, wait_time):
try:
WebDriverWait(conftest.browser, timeout=wait_time).until(EC.visibility_of(element))
return True
except (AttributeError, StaleElementReferenceException, TimeoutException):
return False
Now we can start implementing the Page Object Pattern and first describe a basic frontend page with elements that every Magento page contains or can contain, e.g. header, footer, success and error messages:
class FePage(Page):
@property
def header(self):
return self.find_element(".page-header")
@property
def footer(self):
return self.find_element(".page-footer")
@property
def success_message(self):
return self.find_element("#maincontent > div.page.messages div.message-success")
@property
def error_message(self):
return self.find_element("#maincontent > div.page.messages div.message-error")
Now we can always check if the header and footer are present on the page. The essayist tests for a homepage are:
from common.pages import FePage
from common.pages import is_displayed
def test_home_page_header(browser, config_host, wait_time):
home_page = FePage(browser, config_host, wait_time)
home_page.load()
assert is_displayed(home_page.cms_home_block, wait_time), ".cms_home_block not found"
def test_home_page_footer(browser, config_host, wait_time):
home_page = FePage(browser, config_host, wait_time)
home_page.load()
assert is_displayed(home_page.footer, wait_time), "Footer is not found"
The next step is to describe other pages and their specific properties and methods. For example, if we talk about the Login page, the Username/Email and Password fields and the Login button are specific to this page only:
class LoginPage(FePage):
def __init__(self, browser, host, wait_time):
self.browser = browser
self.host = host
self.url = LOGIN_URL
self.body_class = "customer-account-login"
self.wait_time = wait_time
@property
def email_field(self):
return self.find_element("#email")
@property
def pass_field(self):
return self.find_element("#pass")
@property
def login_button(self):
return self.find_element("button.action.login")
@property
def remind_link(self):
return self.find_element(".remind")
As well action login can be done also only on this page this method is a part of the page:
def fill_and_submit_login_form(self, user_email, user_pass):
try:
self.email_field.clear()
self.email_field.send_keys(user_email)
self.pass_field.send_keys(user_pass)
self.login_button.click()
return True
except (NoSuchElementException, ElementNotInteractableException, AttributeError):
return False
Now we can implement the next tests to check elements on the log in page and its functionality:
from common.pages import LoginPage
from common.pages import is_displayed
def test_login_page_email_field(browser, config_host, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
assert login_page.email_field is not None, "Email field not found"
def test_login_page_password_field(browser, config_host, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
assert login_page.pass_field is not None, "Password field not found"
def test_login_page_login_button(browser, config_host, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
assert login_page.login_button is not None, "Login button field not found"
def test_login_with_incorrect_password(browser, config_host, registered_user, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"] + "make it incorrect")
assert is_displayed(login_page.error_message, wait_time), "No error message on incorrect password"
The registered_user transferred to the test is a fixture. More about it you can read here: 
And on the next attempt we will use the correct email and password and check that the success message appeared as well as customers menu in the header became available instead of the welcome label:
def test_login_with_correct_password(browser, config_host, registered_user, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"])
assert is_displayed(login_page.success_message, wait_time), "No success message on log in with correct password"
assert not is_displayed(login_page.error_message, wait_time), "An error message on log in with correct password"
assert login_page.is_customer_logged_in(), "Customer menu in header after login not found"
In the last example we have a few expected results after only one action. In case if any assert is failed, then the test immediately will be marked as failed and finished, so actions and asserts after that will not be done, in our example it means that if success message was not displayed, but the user is logged in, we will not know that as the last assert will not be checked. To check a few conditions independent from each other we use delayed assert, which allows us to collect expectations and assert all of them at once. So the test will be refactored in this way:
def test_login_with_correct_password(browser, config_host, registered_user, wait_time):
login_page = LoginPage(browser, config_host, wait_time)
login_page.load()
login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"])
expect(is_displayed(login_page.success_message, wait_time), "No success message on log in with correct password")
expect(not is_displayed(login_page.error_message, wait_time), "An error message on log in with correct password")
expect(login_page.is_customer_logged_in(), "Customer menu in header after login not found")
assert_expectations()
This approach makes it possible to describe any page-specific elements and actions and build any complex workflows and user journeys.
More about the Page Object pattern in pytest please check here:
Reporting
After all tests are done pytest by default will output only errors log. To get more information about tests flow and possible present results to managers and customers we also add Allure reports. Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form but allows everyone participating in the development process to extract a maximum of useful information from the everyday execution of tests.
By default Allure will display all tests with output and status from pytest. By adding a single code line to a @pytest.mark.hookwrapper we can automatically save any data on every test finish, it is also possible to save a screenshot even if tests were running in headless mode. In the next example we save screenshot, URL and HTML of the page, saving the state at the moment of the end of the test:
@pytest.mark.hookwrapper
def pytest_runtest_makereport(item):
outcome = yield
report = outcome.get_result()
if report.when == 'call':
request = item.funcargs['request']
driver = request.getfixturevalue('browser')
allure.attach(driver.get_screenshot_as_png(), attachment_type=allure.attachment_type.PNG)
allure.attach(driver.page_source, attachment_type=allure.attachment_type.TEXT)
allure.attach(driver.current_url, attachment_type=allure.attachment_type.TEXT)
In the end we get the next report with saved data to each test, which makes it easier to validate and debug tests:
Allure gives you a variety of options for sorting test cases and reviewing the timeline. With simple @allure annotations, you can specify user-friendly test titles, severity, how the test is split between steps, and many other possible customizations.
Conclusion
As described in the beginning, we wanted to have a tool that checks every MR if it can affect the general functionality of the store, because when we make several changes at the same time on a server, it is sometimes hard to figure out why exactly a certain functionality stops working. For this reason:
- We have developed our own smoke tests that go through both the main user journeys and the admin workflow
- We run these tests automatically on every push and do not merge any MR until all tests have passed
- We reconcile the tests in the same MR if any FE-influenced changes have been made
- After each run, we have a well-structured visual report created with Allure
- we also run these tests after all MRs have been merged into the main branch and are ready to be deployed to ensure one more time that these changes also match and do not cause any problems.
Photo by Andrea De Santis on Unsplash