UI Test Automation in Adobe Commerce


Author Photo
Anna Shepilova
Senior QA Manager

UI Test Automation in Adobe Commerce

Automated testing is an important part of any software development cycle. When testing UI components, everyone has experienced unexpected errors after deployment. Sometimes it's affected functionality that wasn't thought through, sometimes it's specific cases in a database, and many other possible reasons.

This is why automated testing is an important part of any software development cycle. When testing UI components, it is highly beneficial to run a smoke test after every deployment, or better yet, after every merge request (MR) to make sure the reason for an issue is present. If you work with Adobe Commerce (Magento), you've probably heard of the Magento Functional Testing Framework (MFTF), which can also be integrated into your CI/CD process.

MFTF provides a large list of tests to check a variety of settings in different configurations and act on the front-end (FE).  Unfortunately, it also has a downside: the tests are created for vanilla Magento and it can take some time to adapt them to your project, especially if you use a lot of extensions. 

So, in order to be more flexible and have tests that fit our projects exactly, we decided to develop our own UI tests from scratch. 


Architecture

Pytest is a Python testing framework that can be used to create all types of tests, from simple to complex and scalable test cases. Pytest can be easily combined with Selenium to implement the Page Object pattern. Page Object is a design pattern that has become popular in test automation to improve test maintenance and reduce code duplication.

A page object is an object-oriented class that serves as an interface to a page in your AUT. Tests then use the methods of this page object class when they need to interact with the user interface of that page. The advantage is that when the user interface of the page is changed, the tests themselves do not need to be changed, only the code within the page object. Then, all the changes to support the new user interface are in one place.

The Page Object design pattern offers the following advantages:

  • There is a clean separation between test code and page-related code such as locators (or their use if you are using a UI map) and layout.
  • There is a single repository for the services or operations provided by the page, rather than having those services scattered across tests. 

The pattern fits perfectly on a website based on Magento as it usually has highly specialized pages with fixed and specified elements, e.g. a product page usually has price, quantity and add to cart button that are not displayed on any other page, on the other hand email/login and password fields are displayed only on the login page.

Nevertheless, all pages have some common elements like header, footer and messages. These common elements can be described in the main Page class and page-specific elements in inherited classes. 

Before we describe a page interface, we need to start the web driver and call the page. For this we implement the next constructor and the basic methods that can be used on each page:

from abc import ABC
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver

class Page(ABC):
    browser: RemoteWebDriver

    def __init__(self, browser, host, wait_time):
        self.browser = browser
        self.host = host
        self.url = ""
        self.body_class = ""
        self.wait_time = wait_time

    def load(self):
        self.browser.get(self.host + self.url)

    def is_current_page(self):
        if self.body_class != "":
            return self.find_element("body.{}".format(self.body_class))
        else:
            return True

    def find_element(self, selector):
        try:  
             return self.browser.find_element_by_css_selector(selector)
        except (NoSuchElementException, ElementNotInteractableException):
            return None

A very good tutorial on how to create UI tests with Pytest and Selenium can be found here:  Web UI Testing Made Easy with Python, Pytest and Selenium WebDriver - TestProject

In this article I will only explain how to apply the approach described in the tutorial to the needs of Magento applications. In short words here "browser" - is an object of WebDriver (we can support different WebDriver and run our tests on different browsers each time, which are set up in the json file). 

We also implement the most commonly used method that checks if an element is displayed on the current page: 

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

def is_displayed(element, wait_time):
    try:
        WebDriverWait(conftest.browser, timeout=wait_time).until(EC.visibility_of(element))
        return True

    except (AttributeError, StaleElementReferenceException, TimeoutException):
        return False

Now we can start implementing the Page Object Pattern and first describe a basic frontend page with elements that every Magento page contains or can contain, e.g. header, footer, success and error messages:

class FePage(Page):
    @property
    def header(self):
        return self.find_element(".page-header")
    @property
    def footer(self):
        return self.find_element(".page-footer")

    @property
    def success_message(self):
        return self.find_element("#maincontent > div.page.messages div.message-success")

    @property
    def error_message(self):
        return self.find_element("#maincontent > div.page.messages div.message-error")

Now we can always check if the header and footer are present on the page. The essayist tests for a homepage are:

from common.pages import FePage
from common.pages import is_displayed


def test_home_page_header(browser, config_host, wait_time):
    home_page = FePage(browser, config_host, wait_time)
    home_page.load()
    assert is_displayed(home_page.cms_home_block, wait_time), ".cms_home_block not found"


def test_home_page_footer(browser, config_host, wait_time):
    home_page = FePage(browser, config_host, wait_time)
    home_page.load()
    assert is_displayed(home_page.footer, wait_time), "Footer is not found"

The next step is to describe other pages and their specific properties and methods. For example, if we talk about the Login page, the Username/Email and Password fields and the Login button are specific to this page only:

class LoginPage(FePage):
    def __init__(self, browser, host, wait_time):
        self.browser = browser
        self.host = host
        self.url = LOGIN_URL
        self.body_class = "customer-account-login"
        self.wait_time = wait_time

    @property
    def email_field(self):
        return self.find_element("#email")

    @property
    def pass_field(self):
        return self.find_element("#pass")

    @property
    def login_button(self):
        return self.find_element("button.action.login")

    @property
    def remind_link(self):
        return self.find_element(".remind")

As well action login can be done also only on this page this method is a part of the page:

def fill_and_submit_login_form(self, user_email, user_pass):

        try:
            self.email_field.clear()
            self.email_field.send_keys(user_email)
            self.pass_field.send_keys(user_pass)
            self.login_button.click()
            return True

        except (NoSuchElementException, ElementNotInteractableException, AttributeError):
            return False     

Now we can implement the next tests to check elements on the log in page and its functionality:

from common.pages import LoginPage
from common.pages import is_displayed


def test_login_page_email_field(browser, config_host, wait_time):
    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()
    assert login_page.email_field is not None, "Email field not found"

def test_login_page_password_field(browser, config_host, wait_time):
    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()
    assert login_page.pass_field is not None, "Password field not found"

def test_login_page_login_button(browser, config_host, wait_time):
    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()
    assert login_page.login_button is not None, "Login button field not found"

def test_login_with_incorrect_password(browser, config_host, registered_user, wait_time):
    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()
    login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"] + "make it incorrect")
    assert is_displayed(login_page.error_message, wait_time), "No error message on incorrect password"

The registered_user transferred to the test is a fixture. More about it you can read here: About fixtures — pytest documentation Let's say it is a magic that makes some needed pre-actions before test run or defines specific values. The point is that we can create positive as well as negative tests, in this case we checked that on some incorrect password an error message is displayed.

And on the next attempt we will use the correct email and password and check that the success message appeared as well as customers menu in the header became available instead of the welcome label:

def test_login_with_correct_password(browser, config_host, registered_user, wait_time):
    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()

    login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"])

    assert is_displayed(login_page.success_message, wait_time), "No success message on log in with correct password"
    assert not is_displayed(login_page.error_message, wait_time), "An error message on log in with correct password"
    assert login_page.is_customer_logged_in(), "Customer menu in header after login not found"

In the last example we have a few expected results after only one action. In case if any assert is failed, then the test immediately will be marked as failed and finished, so actions and asserts after that will not be done, in our example it means that if success message was not displayed, but the user is logged in, we will not know that as the last assert will not be checked. To check a few conditions independent from each other we use delayed assert, which allows us to collect expectations and assert all of them at once. So the test will be refactored in this way:

def test_login_with_correct_password(browser, config_host, registered_user, wait_time):

    login_page = LoginPage(browser, config_host, wait_time)
    login_page.load()

    login_form_submit_success = login_page.fill_and_submit_login_form(registered_user["email"], registered_user["password"])

    expect(is_displayed(login_page.success_message, wait_time), "No success message on log in with correct password")
    expect(not is_displayed(login_page.error_message, wait_time), "An error message on log in with correct password")
    expect(login_page.is_customer_logged_in(), "Customer menu in header after login not found")

    assert_expectations()

This approach makes it possible to describe any page-specific elements and actions and build any complex workflows and user journeys.

More about the Page Object pattern in pytest please check here: Develop Page Object Selenium Tests Using Python - TestProject


Reporting

After all tests are done pytest by default will output only errors log. To get more information about tests flow and possible present results to managers and customers we also add Allure reports. Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form but allows everyone participating in the development process to extract a maximum of useful information from the everyday execution of tests.

By default Allure will display all tests with output and status from pytest. By adding a single code line to a @pytest.mark.hookwrapper we can automatically save any data on every test finish, it is also possible to save a screenshot even if tests were running in headless mode. In the next example we save screenshot, URL and HTML of the page, saving the state at the moment of the end of the test:

@pytest.mark.hookwrapper

def pytest_runtest_makereport(item):
    outcome = yield
    report = outcome.get_result()

    if report.when == 'call':
        request = item.funcargs['request']
        driver = request.getfixturevalue('browser')
        allure.attach(driver.get_screenshot_as_png(), attachment_type=allure.attachment_type.PNG)
        allure.attach(driver.page_source, attachment_type=allure.attachment_type.TEXT)
        allure.attach(driver.current_url, attachment_type=allure.attachment_type.TEXT)

In the end we get the next report with saved data to each test, which makes it easier to validate and debug tests:

Allure Reporting Suite

Allure gives you a variety of options for sorting test cases and reviewing the timeline. With simple @allure annotations, you can specify user-friendly test titles, severity, how the test is split between steps, and many other possible customizations.

Conclusion

As described in the beginning, we wanted to have a tool that checks every MR if it can affect the general functionality of the store, because when we make several changes at the same time on a server, it is sometimes hard to figure out why exactly a certain functionality stops working. For this reason:

  • We have developed our own smoke tests that go through both the main user journeys and the admin workflow
  • We run these tests automatically on every push and do not merge any MR until all tests have passed
  • We reconcile the tests in the same MR if any FE-influenced changes have been made
  • After each run, we have a well-structured visual report created with Allure
  • we also run these tests after all MRs have been merged into the main branch and are ready to be deployed to ensure one more time that these changes also match and do not cause any problems.

Related Articles

  • 2022-05-31
    Reply named Adobe Solution Partner PLATINUM
    Author Photo
    Martin Altmann
    Adobe Practise Lead
    We are pleased to announce that Adobe has recognised Reply as a Solution Partner Platinum.
  • 2022-05-10
    Advanced routing using Fastly CDN
    Author Photo
    Alexandr Tihanov
    Senior Developer at comwrap
    Fastly is a Varnish-based service used in Adobe Commerce Cloud (Magento) for content delivery, security, and image optimization. In this article, I describe some of the advanced routing features for developers.
  • 2022-04-22
    Disaster Recovery in an Adobe Commerce Cloud environment
    Author Photo
    Alex Lyzun
    Lead Developer Adobe Commerce
    Adobe Commerce Cloud has an automated backup process by design. So, if you act quickly after a notice, you have a good chance to restore your system almost completely.