Test automation is a complex topic where diverse approaches often lead to confusion and misalignment among teams. The Test Automation Roadmap introduces a structured model to help teams, departments, and organizations navigate this terrain effectively.
Why a new model?
I have worked and talked with many fellow testers. One thing that stands out to me is that I often think very differently about test automation. At the same time, there are others who think like me. At first, I thought this was due to a difference in (technical) knowledge or experience. I was wrong. I observed this difference between testers with similar levels of knowledge, similar levels of experience, and very different levels of experience.
Eventually, I realized that this is a mindset difference. It’s obvious in hindsight. Of course, people approach a big complex topic like test automation in different ways. However, it seems that nobody talks about these different ways of thinking. There are plenty of resources about the non-automated testing mindset versus the test automation mindset, but none on mindsets within test automation.
Test automation mindsets
How someone approaches test automation is primarily guided by their mindset. Because of this, mindsets are the foundational concept of the Test Automation Roadmap. The model consists of five valid but very different mindsets:
- First steps 👣
- Cherry-pick 🍒
- Everything 🚀
- Multi-team 🤝
- Enterprise 🌍
Within teams, these mindsets can help you understand how team members think. Team members likely have varying mindsets, which can cause test delays and friction between team members. The shared definitions of the model can help team members to better understand each other. It can also help them to adopt a different mindset when required (and desired). When it’s time to make tests, there is guidance on tool and technique selection that works with your mindset.
Across teams, these mindsets can help you understand how teams use test automation in their workflows. The shared definitions help you communicate with (other) teams effectively about test automation. This way, the model can be used to gain insight into how teams automate their tests.
You can also use the model as a strategic tool that answers test automation questions like: What is your current situation? Where do you want to end up? What do you need to get there? Using the model this way gives you a high-level Test Automation Roadmap for your team(s) backed by shared understanding and guidance on implementation.
The simplified model
The model is a kind of graph. This is clearer in the full model, but we start at the beginning with the simplified model.
On the horizontal axis, we have the test scope. It tells us how many tests exist compared to all tests that can possibly exist in our test suite. More to the right means more tests and more to the left means fewer. In a test suite at the right edge, you can’t add a single test that will add any value. A test suite at the left edge contains no tests.
On the vertical axis, we have the organization scope. It tells us how much of the organization is involved in the test suite. Practically speaking, this axis consists of three layers:
- Top layer: Multiple departments
- Middle layer: Multiple teams but within a single department
- Bottom layer: Within a single team
In a test suite at the top edge, literally everyone in the organization is somehow involved. At the bottom edge, only a single person is involved.
The model does not plot teams or applications. Instead, we plot our test suites on these axes. An application or team can and should have multiple test suites.
First steps mindset 👣
You want to, or have to, prove that test automation will offer value for your situation. Maybe you have never made an automated test before. Maybe the team has never worked with them. In any case, you must convince someone that it’s worth investing time and energy in test automation.
Cherry-pick mindset 🍒
You already know that test automation will be valuable in your situation. You automate low-hanging fruit and high-risk features. You only automate if it’s worth it.
Everything mindset 🚀
You automate all your tests unless it takes too much effort due to technical reasons. You only test manually as a last resort.
Multi-team mindset 🤝
You automate chain tests in (relatively) small shared environments. This is not organization-wide end-to-end testing but testing with your neighbors. You try to include as few teams in the shared environment as possible. The more teams you add to the shared environment, the fewer tests you make.
Enterprise mindset 🌍
You automate a small but crucial part of quality control for multiple departments or the entire (global) enterprise organization. With these tests, you work on compliance at scale. You generally focus on security compliance, self-imposed compliance, and law compliance.
The full model
The full model is a more nuanced version of the simplified model above. This version highlights many of the things I talked about in the individual mindsets.
Please note that the vertical axis uses a logarithmic scale.
Using the model
In the model, we plot our test suites. To explain how this works, I asked a team to describe their situation, resulting in a real-world example of a backend team.
This team has some unit tests that only developers are involved with. They are unit testing with the cherry-pick mindset, leaving the majority of test coverage to the component and integration tests.
The integration tests cover more functionality, and the team wants to expand their integration testing efforts. In other words, they want to move right in the model. Moving right also means they must move up to prevent individual team members from becoming a single point of failure.
Their component tests cover the most functionality. Almost everyone on the team is involved with these tests in some way. Most likely, the only ones not involved are the non-technical team members (for example, product owner, scrum master, business analyst).
The final test suite is neighbor testing, though they have not automated these tests yet. They do have a manual test set that they will automate soon.
Conclusion
The Test Automation Roadmap is a new model to think about how to position and approach test automation in your team, department, and organization. In the model, you plot your test suites. This allows you to visualize where your test automation is at right now and where you want to go. The five mindsets also offer guidance on how to reach your goals.
Each of the five mindsets is a valid but different way of thinking about test automation.
- First steps 👣 — Prove that test automation works for your situation
- Cherry-pick 🍒 — Automate low-hanging fruit and high-risk features
- Everything 🚀 — Automate everything unless it takes too much effort due to technical reasons
- Multi-team 🤝 — Automate chain tests with your neighbors
- Enterprise 🌍 — Compliance at scale
How will you use the model?