Maria Homann
At the heart of agile software development is a concept that's stood the test of time, acting as a trusted guide for developers and testers: the Testing Pyramid. In this article, we help you understand the original testing pyramid and reshape it slightly to fit the 2024 testing landscape.
Skip to:
The evolution of the testing pyramid
Understanding the testing pyramid in a modern context
The evolution of the testing pyramid
A concept coined by Mike Cohn in his book Succeeding with Agile and refined in Ham Vocke’s article The Practical Test Pyramid, this isn’t just about tests. It’s a blueprint for building resilient software that can stand up to the rigors of real-world use.
But even with such a solid foundation, many struggle to put these ideas into practice.
In this article, we're taking a fresh look at the testing pyramid, dusting off the cobwebs, and diving into how it can be applied in today's fast-paced development environments.
What we propose is a new testing pyramid, one where the top takes greater precedence. This model emphasizes testing from the perspective of the user, concentrating on their experience rather than that of the developers.
This is not to undermine the importance of early testing disciplines, such as unit testing, but to slightly shift the focus.
Why this shift? In an increasingly digital world, nothing is more crucial than the customer/user journey. Developers can create countless unit tests to verify every detail, yet users might encounter issues in seconds, often using the product in ways not intended by its creators.
It’s not enough to follow predefined "happy paths" – the real world is messier; it’s more complex, and more unpredictable.
With a wealth of smart, user-friendly test automation tools now at our fingertips, crafting and maintaining end-to-end tests that truly reflect user needs isn’t just a possibility; it’s crucial and game-changing.
What is the testing pyramid?
The first testing pyramid, introduced by Mike Cohn in 2009, was a metaphor for the ideal distribution of automated tests in software development, emphasizing a large base of unit tests and fewer high-level tests. It is sometimes also referred to as the test automation pyramid.
This original version of the pyramid illustrates the proportion and relationship of different kinds of tests that contribute to a well-balanced and effective testing strategy.
The layers from bottom to top are:
Unit tests
- What are unit tests: These are at the base of the pyramid and should, according to Mike Cohn, make up the majority of your tests. Unit tests are small and fast tests that check individual functions or methods in isolation from the rest of the system. They are intended to be simple and quick to write and execute, allowing developers to quickly identify and fix errors in the foundational elements of the application.
- Who performs unit tests: Unit tests are primarily performed by software developers. It is their responsibility to ensure that their code operates as expected, embodying a proactive approach to quality assurance right from the developmental stage. This practice underscores a paradigm where developers are not just creators but also the first line of defense in ensuring software reliability.
- When is unit testing performed: Unit testing is an integral part of the development phase, serving as a continual backdrop to the coding process. It is executed concurrently with the development of features, ensuring that each unit of code is tested in isolation immediately after its creation. This methodology promotes a culture of immediate feedback and iterative improvement, pivotal for agile development environments where speed and adaptability are key.
Integration tests
- What are integration tests: Sitting above unit tests, integration tests verify that different modules or services work together as expected. These tests are fewer in number compared to unit tests because they are more complex and time-consuming to set up and run. Integration tests ensure that the connections and interactions between modules, external services, or systems work correctly.
- Who performs integration tests: Integration tests are typically carried out by specialized testers. This delineation of responsibilities ensures that while developers focus on the intricacies of individual units, testers can concentrate on the broader perspective of how these units interact. This division of labor is crucial for maintaining a high standard of quality, as it allows for a more focused approach to each level of testing.
- When is integration testing performed: Integration testing takes place in the aftermath of the development phase but prior to the software's release. This strategic timing ensures that all individual components, having been verified through unit testing, are scrutinized for their collective operation. This step is crucial for catching issues that arise not within the isolated units, but in the interaction between them, safeguarding the integrity of the final product.
End-to-End Tests (E2E)/UI Tests
- What are end-to-end tests: At the top of the pyramid, these tests simulate real user scenarios from start to finish. They are the most comprehensive, covering the entire application to check if all parts work together as expected from a user's perspective. These tests are the most complex and can require more time and resources to write, maintain, and execute. Thus, they are fewer in number compared to unit and integration tests.
- Who performs end-to-end tests: End-to-end tests are typically carried out by QA teams. This specialized group takes the helm after functional and system testing phases, bringing a comprehensive lens to the application's functionality by simulating real user interactions. Their role is pivotal in ensuring that the application not only meets technical specifications but also delivers a seamless and intuitive user experience.
- When are end-to-end tests performed: End-to-end testing is generally conducted after integration testing. This timing is strategic, as it allows QA teams to assess the application in its entirety, ensuring all components work harmoniously before the software is released to the public. This phase is crucial for validating the overall system's performance, stability, and reliability from an end-user's perspective.
Since its introduction, the testing pyramid has acted as a guiding principle for teams to efficiently distribute their testing efforts.
At the core, it encourages a high volume of low-level tests, which tend to be quick and cheap to execute, and a more selective approach to high-level tests, which tend to be slower and more expensive.
This strategy aims to achieve an ideal mix of test coverage, speed, and cost efficiency.
Over the years, however, the model has faced criticism for its simplicity; What about the role of manual testing? Aren’t UI tests more important today?
The model arguably simplifies things. But maybe its simplicity is actually its strong suit, making it an easy to remember and handy rule of thumb for development and testing.
Understanding the testing pyramid in a modern context
In today’s software development space, however, things are moving quickly, and test automation tools are widely available and adopted, which has transformed how testing is done.
Running end-to-end tests isn’t necessarily as complex and time-consuming as it used to be.
Ham Vocke wrote in his 2018 description of the practical test pyramid that “End-to-End tests come with their own kind of problems. They are notoriously flaky and often fail for unexpected and unforeseeable reasons.”
The reason behind, he says, is that “ The more sophisticated your user interface, the more flaky the tests tend to become. Browser quirks, timing issues, animations and unexpected popup dialogs are only some of the reasons that got me spending more of my time with debugging than I'd like to admit.”
As a solution, he recommends writing as few end-to-end tests as possible. He also recommends that end-to-end tests should be written with a focus on the most important steps of the user journey.
As an example, he writes “If you're building an e-commerce site your most valuable customer journey could be a user searching for a product, putting it in the shopping basket and doing a checkout. That's it. As long as this journey still works you shouldn't be in too much trouble. Maybe you'll find one or two more crucial user journeys that you can translate into end-to-end tests. Everything more than that will likely be more painful than helpful.”
While this user-centric approach and the given example definitely still stands, a lot has happened since Vocke wrote the article, making his last statement “Everything more than that will likely be more painful than helpful” a bit less true.
In 2018, most test automation tools on the market were open source and code-based, and required at least some level of programming experience to use. This meant that your developers or specialized test automation engineers would have to get involved in the end-to-end layer of the testing pyramid. Not only for test creation, but also for test maintenance. This approach works for some businesses, but it doesn’t scale.
What’s more, tests at this time would tend to fail a lot more because of things like dynamic elements in the DOM, or timing issues, as Vocke also describes.
Today, however, in the age of AI, self-healing capabilities are becoming more common in test automation tools. Combine this with a wide range of in-built intelligence mechanisms and general technology advancements, and you have businesses who are now able to build highly stable and robust UI-based test automation.
As a result, you’re able to build, maintain, and scale your end to end tests to cover much more than just a handful of user journeys.
– That said, there’s always a balance to strike with automated tests, and you definitely shouldn’t have more end-to-end tests than unit tests. But it’s probably safe to say that the tip of the pyramid has widened.
The new testing pyramid
So where does that leave us?
More recent attempts to redefine, or shall we say reshape, the testing pyramid have involved:
- Downsizing the layer of unit tests, turning it into a testing diamond
- Reemphasizing the importance of manual testing, turning it into a testing ice cream cone
- Flipping the pyramid, turning it into a testing pizza
- And finally extending the body of the model with actual limbs, turning it into a testing crab
Perhaps the take-away is that we shouldn’t care so much about the shape. The testing pyramid is still a useful way of understanding the hierarchy of testing. As long as it’s seen through a modern, user-centric lens and with acknowledgement that the software testing space has changed dramatically in recent years - and will continue to change.
But we figured we’d give the reshaping thing a try anyway:
This model maintains the layer of unit testing, which provides a foundational quality check. Integrations also keep their prominence. But end-to-end tests take up more space, both in terms of numbers and in terms of importance.
We’ll call it the testing hourglass.
The new testing pyramid: a real-world example
Today, businesses can do much more end-to-end testing thanks to robust and reliable software testing tools.
Below is an example of a business that did this. Mattress Firm, a large US retailer, landed on a less-than-ideal test automation solution that wasn’t giving them reliable results. It was more painful than helpful, to use Vocke’s words, and they ended up resorting to manual testing.
Mattress Firm were then introduced to a more user-friendly test automation solution that gave them the stability and scalability they needed to implement end-to-end tests.
As a result, they were able to improve their customer journeys and enhance their e-commerce experience.
Watch the story here:
Continue learning
Are you in the process of building out your test automation strategy? Then make sure to get our test automation strategy checklist. It takes you through all the steps you need to know to create an efficient and effective testing strategy.
