Writing a test is easy, ensuring your project is tested, is hard. This post is here to advocate for end-to-end testing, to make controversy mainstream and hopefully convince you to join the cause.
At the Android Summit, Donn Felker shared his views on ‘testing for success in the real world’. Advocating for end-to-end tests over unit tests, and that this viewpoint is controversial. The following is a summary outline of the talk:
Does that code you just wrote actually work? How do you know? How do your teammates know? You did write tests for that, right? Unit? Integration? End-to-End? You only changed two lines of a legacy piece of the app… but still, how do you know this didn’t break anything?
Let’s face it testing your application is difficult and tedious. Where can you get the most bang for your buck? What’s the 20% of work that gets you 80% of the return? In this session you’ll learn where you can focus your attention to gain the most traction in your testing endeavors. Juxtaposing the benefits of unit testing vs end-to-end testing.
Spend 60% of your efforts, and thoughts on end-to-end tests, and the remaining 40% on the rest.
You can find it on YouTube here: Testing for Success in the Real World - Donn Felker.
Therefore the TLDW paraphrase position being: “The testing pyramid encourages a 70/20/10% test number split, and I don’t like that focus on the number of tests. Ask yourself where are you going to get the most bang for your buck, and that’s in the end, end-to-end tests. Adopt a different end-to-end mindset, end-to-end tests do mimic real world usage, that’s real users. Your users don’t care that your test suite passes, your users care that your application works.”
End-to-end tests really are the closest thing we have to having a user sitting next to us using the app. End-to-end tests capture the user behaviour, they show all the pieces connected together and working. If you think of a timing analogy; end-to-end tests are the runtime-problem highlighter, whilst unit tests are more compile-time, or another analogy, unit tests are the canary in the coal mine for development and end-to-end tests are the fireguard whilst burning the coal at home.
The testing pyramid reflects a notion of quantity, but we want to be thinking of the quality of our tests. The pyramid encourages good development practices for the individual developer, but doesn’t highlight anything of the complexity of time/effort needed to go into each layer (unit, system, e2e). How can we refactor this testing pyramid to reflect what it is needed from our apps test suite?
Developers are always in a rush to deliver, whether this is a self imposed idea of ‘being in the zone’ and churning out code, or whether this is management/company imposed, with pressure of deadlines etc.
The amount of effort to get a unit test up and running vs an end-to-end test on Android is minimal. Espresso and other acceptance testing technologies require due diligence, patience and a cognitive shift away from the individual lines of code being written. This isn’t an excuse, but another reason why developers will prefer to unit test, this leads to the “oh sh*t” scenario of testing the pieces but never the whole.
Projects start with a single line of code, there is a setup of the project and the overhead of the gradle build system, getting it deploying on a real device etc. but the project isn’t “your own” until you write that first line of bespoke code. You could, end-to-end test this line of code. Have Espresso start your app and verify it is open (let’s say with a blank white UI). You could, unit test this line of code and validate it’s what you intended.
Each of these two scenarios have a wildly different build time (as mentioned in the second last paragraph), but the point here is, at the start of the project the UI is constantly changing. The UI will be getting feedback from other departments in the company (your code architecture won’t), and so the UI is changing. This is a discouragement to start your project with end-to-end tests. They’ll be red - a lot. That takes effort. Whilst it’s the right thing to do, and very important to have that end-to-end scenario setup and tested as soon as possible. Here I am highlighting the (conscious or not) amount of effort involved and why developers (conscious or not) do not jump straight to end-to-end tests.
Or written differently, when you first start a new project, you have no UI, once you do, the UI is still very malleable and not set. This is a reason why end-to-end tests may be avoided, having them at the start increases overhead & flakiness, but if they aren’t their from the start, then the pattern/process is not in place so they won’t arrive later. Without active effort.
Another way to look at the pyramid is through the lense of the Pareto principle, this states:
The Pareto principle (also known as the 80/20 rule, the law of the vital few, or the principle of factor sparsity) states that, for many events, roughly 80% of the effects come from 20% of the causes.
Mathematically, the 80/20 rule is roughly followed by a power law distribution (also known as a Pareto distribution) for a particular set of parameters, and many natural phenomena have been shown empirically to exhibit such a distribution.
One way to interpret this rule is to look at your end-to-end tests as that vital 20%. 20% of your testing efforts will catch and highlight 80% of your bugs.
Potentially your current application testing pyramids look like the following (aka missing the end-to-end tests):
You will be missing 80% of your bugs! The Pareto rule is a strong indicator to the importance of end-to-end tests.
The testing pyramid as a concept is usually applied to your whole application. It’s interpreted as how your application should be tested and this can give the effect of “someone else's problem”. If you are only working on a small part of an application, say the login feature. You may (consciously or not) consider the idea of end-to-end testing as outside of the scope of your work - an application wide issue, not worth the time when just writing the login.
A way to counteract this thought process is to break the pyramid down, talk about the testing pyramid at the module level, feature level, sprint level or whatever smaller unit of development/time that makes sense for you and your team.
Finally, as has been advocated throughout this post, consider the payoff of the tests you are writing. When wanting to highlight the user impact, perhaps a one-dimensional pyramid is not the best way to discuss testing, but a multi-dimensional representation is needed that also considers users.
How about one of the following:
- Consider the weighting of who gains the most from these test:
- Consider who would be impacted earliest when these tests break:
In conclusion, the testing pyramid doesn’t have an axis for importance, or the value driven from the different testing layers. End-to-end tests are an important building block of your application, a pyramid without the top portion is not a pyramid at all! Remember the Pareto principle and that 80% of your value will be derived from 20% of your tests. Always, measure, learn and adapt to ensure the testing pyramid works for you and your project.
Pareto Principle in action:
- 80% of the quality of your software comes from 20% of your tests
- 80% of your effort will create 20% of your tests
- 80% of your bugs are caught by 20% of your tests
- 80% of your tests are redundant
The 20% is concentrated in the top of the pyramid.