The Test Automation Roadmap is a model that splits the complex topic of test automation into five mindsets. This article covers the Everything mindset. This article is part of a bigger story on the Test Automation Roadmap. You should start with the introduction.
Automate everything, unless
When you have the Everything mindset, you automate all your tests unless it takes too much effort due to technical reasons. It’s an automation-first mindset.
In the Everything mindset, you only automate tests within your team. Testing with other teams is covered in the Multi-team mindset.
The Everything mindset can be controversial for those who don’t have it. Opponents of the mindset often see it as automation for automation’s sake.
Important
With this mindset, you try to automate everything you test. It’s not feasible to do this alone, so you make test automation a fundamental part of the team process. For example, the team agrees that test automation is part of every story. If test automation is not done, the story is not done. No exceptions.
Maintenance becomes important due to the sheer number of tests you make. You have to repeat every bit of maintenance for every test. You put a lot of effort into stories that improve maintainability so you don’t drown in maintenance work in the future. The same rationale applies to test performance. A fast test suite allows you to get fast feedback. It also allows you to iterate faster while making new tests. The team agrees that maintenance and performance stories add business value.
Regression through only test automation must be enough for a release. Sometimes, you may test something manually. However, you only test manually in exceptional situations.
Problems
It takes a lot of time to write and maintain all automated tests. This is again due to the number of tests. Test automation with the Everything mindset requires a long-term investment.
After working with this mindset for a while, every new test will be more technically complex than the last. You need more and more technical knowledge to make the next test. You already added the easy tests. At the same time, every new test is less useful due to the law of diminishing returns. Stay wary of when it’s no longer worth adding tests due to technical reasons like this.
Tools & Techniques
This is the mindset where automation nerds thrive. In other mindsets, you should be wary of using tools and techniques that are too technically complex, but in this mindset complexity is fair game.
The technical side is important, but the Everything mindset breaks down if you only focus on the technical side. For this to work, the entire team must be on board. Pay close attention to the team process side of the equation. Adopting the everything mindset on your own often makes you a single point of failure.
Some general advice:
- Do it together. Explicitly add test automation to your team processes.
- Keep any test as simple as possible. Don’t judge simplicity by your skills but by those of your team.
- Static code analysis is always a good idea.
- When writing unit tests, also consider using more complex approaches. Some examples:
- Property-based testing
- Mutation testing
- Snapshot testing
- Prefer API tests over GUI tests as they are faster and easier to maintain.
- When writing API tests, consider tooling categories like:
- You write tests in a programming language
- You create tests in an application (e.g. Postman, ReadyAPI)
- A tool generates tests for you
- When writing API tests, also consider using more complex approaches. Some examples:
- Property-based testing
- Snapshot testing
- Schema-based testing, or go the extra mile with contract-based testing.
- When writing GUI tests, only use tools where you write tests in a programming language.
- When writing GUI tests, also consider using more complex approaches. Some examples:
- Snapshot testing (or screenshot testing)
- AB testing
- Property-based testing. However, test duration can increase exponentially if you’re not careful.
- Use both continuous integration (CI) and continuous deployment (CD) pipelines.
- Performance tests are fair game if you need them. Keep in mind that MVP performance tests are often enough to find major bottlenecks.
- Basic security tests are fair game. You won’t replace a security expert, but you can test basic things like dependency vulnerabilities, cross-site scripting (XSS), SQL injections, etc. If you tackle the basics, a security expert can focus on the rest.
Position in the model
In the Everything mindset, you automate a lot of tests. If you end up in the white area to the bottom-right of the Everything mindset you create single-point-of-failures. Each team member is responsible for too many tests. Because of this, you will reach a limit on the amount of tests you can add. If you want to add more tests after reaching this limit, use the Everything mindset with more team members to share the load.
While the Everything mindset can reach 100% of all possible tests, this should never be a goal. Reaching 100% is not a good idea due to the law of diminishing returns.
Real-world example
A short story follows about how a team used the Everything mindset in a company of about 45000 employees (according to LinkedIn). The team made an internal tool with a front-end and back-end that implemented complex business logic. This logic being correct was important to the success of the entire organization. Due to the complex nature of the business logic, they used the Everything mindset to test the application.
The team members added test automation to their definition of done, and they kept to it. A story was not done if test automation was not complete. They also wrote their test automation in the same Git branch the developers used to implement the story, making test automation an integral part of the team process.
The teams’ testers made API tests and GUI tests. The API tests tested the business logic of the backend. The GUI tests had a much smaller scope to test the integration between the front-end and back-end. They used Jenkins pipelines with pull request triggers and merge restrictions that enforced that all tests must pass. In other words, they used a full continuous integration approach to testing.
The team faced performance issues caused by the amount of tests and a lack of priority for fixing them. They worked around the performance issues by only running a subset of tests in pull requests. Once a day, the team ran everything in a scheduled pipeline.
The biggest issue the team faced was test maintenance. For this issue, the team never found a good solution.
Conclusion
The Everything mindset is an automation-first approach where you automate all possible tests within your team unless it takes too much effort due to technical reasons. You’ll make a lot of tests with this mindset. With a large number of tests come great maintenance and performance concerns.
This mindset works best if you do it together. Involve the whole team by embedding test automation into your processes.
You always test in isolation. Applications of other teams are out of scope in the Everything mindset.
The Everything mindset can be controversial, but it’s often worth it.