June 4, 2024

Who is using MFTF?

Yireo Blog Post

On LinkedIn, I asked who is using the Magento Functional Testing Framework (MFTF). Few. Here are some of my thoughts contemplating on why this is the case.

First, the Magento Functional Testing Framework

When Magento 2 came out in 2015, there were already a lot of things covered with tests. Magento included unit tests, integration tests, static analysis checks (like PHPCS and PHPStan) and even functional tests with PHPUnit. But navigating the frontend with some browser as a means of testing was not yet included, until MFTF came along.

MFTF bundled Codeception with the Selenium browser automation tool, plus a Magento-specific XML-layer to generate tests in a granular way. Using XML makes a lot of sense: When two Magento modules shape up a certain page, a third party module could be used to change it. With MFTF, the core modules could include XML-based functional tests, while the third party module could simply extend upon this XML, similar to how the third party module was extending the system. Yes, it is XML, but it makes sense because it is extensible.

The complexity is overwhelming. And the performance sucked

MFTF has been difficult to adopt: It's XML includes so many aspects - actors, sections, metadata, pages, data providers and then the real tests - that it is hard to get started with even a simple test like navigating to a certain page. Next to that, the procedure to get started with MFTF has been cumbersome all a long. Every time that I installed it to run it, it took me easily one hour. Usually, I tried to setup a single Magento instance with MFTF installed, to reuse that instance for extension testing over-and-over again.

Next to that, the performance is also bad. Underneath, MFTF uses the Java-based Selenium tool, which used to be great, until other tools like Playwright and Cypress came alone that are way faster. Adding to this, MFTF needs to parse all of the XML files and then generate actual PHP tests, before you can run the tests. It feels all a bit slow.

Running the entire MFTF suite for all core modules took days.

Some history

Years back I was part of ExtDN (which is now in a peaceful passive state), the extension developer network. And there, the role of MFTF was seen as quite important: The only way that extension vendors could claim stability was by focusing upon the interoperability between their module, the core and perhaps other modules. And the only framework flexible enough to allow for this was MFTF.

We worked together with great people from Magento, including Tom Erskine (who maintained and further developed MFTF). For extensions, it made sense.

However, after organizing MageTestFest twice, I saw that MFTF was also being adopted by agencies for custom projects. I always found that strange, even though Magento advocated that MFTF could be used for that purpose too: If you would have a Magento shop with 20 modules and only 1 module would not comply to MFTF already, then it could break the system.

So, who is using MFTF?

The LinkedIn poll showed that 95% of the respondents did not use MFTF. MFTF is not very popular these days. It is almost absent with agencies, it is only used by some extension vendors.

MFTF for whom?

As I have seen it, MFTF was great for extension vendors, but horrible for vendors. MFTF combines tests for the core with tests for third party modules. But as an agency, you don't want to test the core or third party modules, you want to test the customization of the shop. End-to-end tests are therefore much more common with agencies - Cypress, Playwright, etc.

However, with an E2E test, you are only testing the system from the outside. There are still certain parts of the application that you would want to test from the inside: Price calculators, product configurations, checks, validations, filters. For this, functional tests are the way. However, with MFTF, the functional tests are run from the outside (in which they are similar to E2E tests). And I'm referring to the functional tests that are more similar to integration tests.

What is functional testing anyway?

There seem to be multiple definitions of functional testing, but I personally always define it as the assurance that various aspects of the delivered product actually work. From the perspective of Magento (and Magento being the product), MFTF is great for that. If the product is a third party module, an integration test is great. But for custom projects it is either E2E tests or PHPUnit-based tests that are simply run on a copy of the live database.

So, for me, functional tests are very similar to integration tests - they could even include the same code. However, with integration tests, you are taking the core and only a limited subset of modules to see if things are integrating correctly. And a dummy database is handy with that. And with functional tests, you are taking the entire project (with the core and all of the modules installed on top of it). And with that, an actual database makes more sense.

No more MFTF

In the past, I have been explaining people how to use MFTF. However, in real life, nobody used it. Because of this, I'm skipping MFTF in all my trainings and advocating not to use it anymore. There are other better ways.

Posted on June 4, 2024

Do not get started with testing because you have to. Get started with it, because it improves the quality of your code, which leads to less bugs, which causes less stress, which is making you happier.

Checkout our testing training

About the author

Author Jisse Reitsma

Jisse Reitsma is the founder of Yireo, extension developer, developer trainer and 3x Magento Master. His passion is for technology and open source. And he loves talking as well.

Sponsor Yireo

Looking for a training in-house?

Let's get to it!

We don't write too commercial stuff, we focus on the technology (which we love) and we regularly come up with innovative solutions. Via our newsletter, you can keep yourself up to date on all of this coolness. Subscribing only takes seconds.

Do not miss out on what we say

This will be the most interesting spam you have ever read

We don't write too commercial stuff, we focus on the technology (which we love) and we regularly come up with innovative solutions. Via our newsletter, you can keep yourself up to date on all of this coolness. Subscribing only takes seconds.