The Test Automation Roadmap is a model that splits the complex topic of test automation into five mindsets. This article covers the Cherry-pick mindset. This article is part of a bigger story on the Test Automation Roadmap. You should start with the introduction.
Automate if it’s worth it
When you have the Cherry-pick mindset, you already know that test automation will add value to your situation. You automate low-hanging fruit and high-risk features. You only automate if it’s worth it.
In the Cherry-pick mindset, you only automate tests within your team. Testing with other teams is covered in the Multi-team mindset. There is quite a bit of overlap between the two. When talking about the testing mindset, most people use either the Cherry-pick mindset or the Multi-team mindset.
Important
The Cherry-pick mindset focuses on creating maximal value with minimal effort. You automate easy tests, apply risk-based testing, or do both.
How long it takes to make a test is important to you. How long it takes to run tests is not. Why would you optimize your tests if you can make more while it runs?
You consider test maintenance, but you only put in effort while setting up your test suite for the first time.
Problems
Maintenance will probably be your biggest problem. You do consider maintenance when picking tools and setting up your test suite. You also recognize what maintenance you should do, so you write maintenance stories. However, these stories live at the bottom of the backlog and never get priority. Because of this, it can be hard to keep tests passing in the long run.
You have a similar problem with performance. You write stories to make your tests run faster, but they end up at the bottom of the backlog, next to the maintenance stories. Because of this, running your test suite may take hours.
Your automated test suite is not enough for automated regression before a release. It’s a good starting point, so run the automated regression first. After the automated tests, you must also test manually before release.
Tools & Techniques
A great place to start is Risk-Based Testing. Assess your risks and write your tests based on the risk-to-effort ratio. Any test you automate should have a high risk-to-effort ratio.
When you pick tooling, keep maintenance in mind. Don’t pick tools that require a lot of maintenance. For example, you should not use record & playback tools. Also, steer clear of snapshot testing when your application changes regularly.
In the Cherry-pick mindset, you never cross team borders. Any test that runs on a shared test environment (e.g. end-to-end) is out of scope. Testing on shared test environments uses the Multi-team mindset instead.
Some general advice:
- Keep any test you write as simple as possible. Multiple small tests are easier to maintain compared to one big test.
- Steer clear of chain tests and end-to-end tests. In this mindset, you always test your application in isolation. Ask your local neighborhood developer about how to run your application in isolation.
- When you use static code analysis, use an open-source or company ruleset. You can create a custom ruleset, but the discussion that inevitably sparks is not worth it.
- When you write unit tests or component tests, don’t get fancy. There are advanced strategies like mutation testing or property-based testing, but they are not worth it in this mindset.
- Prefer API tests over GUI tests as they are less complex.
- When writing API tests, stick with tools in one of the following categories:
- You write tests in a programming language
- You create tests in an application (e.g. Postman, ReadyAPI)
- A tool generates tests for you
- When writing GUI tests, only use tools where you write tests in a programming language.
- Pipelines are great to run your tests periodically. If you feel fancy, go for a full continuous integration setup instead.
Position in the model
In the Cherry-pick mindset, you won’t automate many tests. Instead, you focus on the ones that add a lot of value. The curved right border of the Cherry-pick mindset shows us that if we want to test more, we should do it together. You can test a lot by yourself, but you’ll reach a limit on the amount of tests you can add in this mindset. If you want to add more tests after reaching this limit, adopt the Everything mindset or use the Cherry-pick mindset together.
Real-world example
A short story follows about how a team used the Cherry-pick mindset in a company of about 2000 employees (according to LinkedIn). In this team, they barely tested anything. They recognized that this was a problem and hired experience — an external tester. They tasked the tester to set up test processes and let teams gain experience with testing.
The new tester introduced a strategy that involved test automation with the Cherry-pick mindset. The tester skipped the First Steps mindset with an authority argument. Soon, they started to set up test automation with Playwright and Specflow. They explicitly choose not to automate all tests but focus on the most important and easily automated features. Tests worth doing but not worth automating were added to the new manual test suite.
They used Specflow to involve more people in the testing process. Specflow was useful due to the lack of experience in the organization and the lack of good documentation. This focus on participation and documentation means that they largely ignored test performance. If they stick with the Cherry-pick mindset, performance will probably be an issue in the future. There was some attention for test maintenance during the tooling setup, but they will probably run into maintenance issues in a few years.
Conclusion
Test automation with the Cherry-pick mindset is about only automating if it’s worth it. Determine worth by risk analysis and ease of automation. In the Cherry-pick mindset, you strive for maximal value with minimal effort. Because of this focus, you may suffer from poor maintainability and performance, especially in older test suites.
You always test in isolation. Applications of other teams are out of scope in the Cherry-pick mindset.
Despite its limitations, the Cherry-pick mindset is a very natural testing mindset. Many people consider it to be a good practice.