Rethinking Pyramids
We as humans have a habit of getting stuck in our ways of thinking. Once we learn something, it is hard to learn away from it. And the longer we are stuck in our ways, the harder it gets. In this blog post we are going to be looking at something very familiar to many of us, and spend some time thinking about it. In the words of John Ruskin: “Quality is never an accident. It is always the result of intelligent effort.” So let’s use our gift of intelligence to rethink for a while. To rethink the pyramids guiding our work.
Rethinking leads to iterative improvement
In this post, I will be talking a lot about rethinking. You may have seen the big headliner on our webpage as well: Rethinking Quality Assurance. But what does rethinking actually mean? I thought the easiest way to approach it is through an example. One from the history of software testing. How software developers and testers have had to rethink their approach to meet the needs of an evolving industry.
First, there was an age of Debugging. Back then testing didn’t have its own identity yet. In the age of debugging it was basically about encountering and fixing bugs while doing development and running your code, instead of actively trying to test.
But quickly software became bigger and more complex, and you had to start specifying how it should work in different scenarios. And people thought: You have to be able to demonstrate that the software works as specified. Thus testing entered the age of demonstration. It became about designing tests based on predefined requirements to prove system functionality, using expected and recognizable parameters. This was also when the number of test cases became a common measurement for quality.
But this way of testing still left a lot of defects uncovered. And as the complexity of the software again kept increasing, testers needed to rethink. Rethink how to find faults where normal testing does not find them. Thus came to be the age of destructive testing. Many testing techniques that are used even today were born in this age. The goal became to demonstrate that the software does not work the way it’s supposed to.
Gradually through trial and error and experimentation, we arrived at the time of evaluation where it became a common practice that, during the lifecycle of a product, testing activities were done and evaluated and quality is measured. Test automation also started to raise its head. But business, testers and developers were still working in their separate siloes. And there was a big change taking place. And that change was known as agile.
Again testers needed to rethink how things would fit into this new way of looking at things. The age of prevention came to be. This is the modern age. This is where we are today. Our goal nowadays is to prevent defects from reaching the product at all. We want to find and eliminate any issues as early as possible with the help of the whole team. Testers are involved in all parts of development, all the way from the beginning. Stuff like Agile Testing and Exploratory testing have become norms known to most testers. And to support the new workflows and fast feedback, continuous testing through automation has become a standard thing.
Preparing for the future
Who knows what the future brings? One thing is for certain: we need to rethink in time so we don’t get caught with our pants down. Maybe the future is all about testing with AI and using machine learning to help us. Or maybe it will be focused on the more humane aspects of quality? Testing against biases. Testing that our systems are inclusive to everyone. Testing the charisma of the product. One thing is for sure: the complexity of systems and software just keeps increasing.
Maybe we even need to rethink the whole meaning of quality
That is what rethinking means to me. It means moving forward. Rethinking is an iterative process of Improving.
So, about those pyramids?
What do pyramids have to do with testing, you might be wondering. Well… I would presume that everyone, who has been working with either test automation, or test strategies has run into the following pyramid. The famous (or infamous) TESTING PYRAMID.
This pyramid is used by almost every organization that uses test automation as part of their toolkit. It tries to define how much of each level of testing should be done. Lots of unit tests because they are fast to create and run, and defects found by them are cheap to fix. A moderate amount of integration tests to verify stuff like APIs and how components work together. Integration tests are usually slower than unit tests, but still faster than E2E testing through UI. And then the least amount of E2E tests through UI. They are generally harder to maintain, slower to run, and find issues so late that they have already become slower to fix.
This pyramid is all about Return On Investment. But only for test automation effort and runtime. It tries to guide us in how we should focus our efforts on test automation so that we get the most value for our runtime.
What if I told you that the pyramid is outdated
Are test runtime and amount of tests on each specific level the things we want to focus on these days? Running tests is cheaper and faster than ever before. This is thanks to easy-to-obtain cloud resources to enable concurrent runs and more efficient testing tools and frameworks.
Furthermore, should we just go for a return on investment? Should we always go with quality first instead?
But what is quality?
One could argue that Quality is what the users value. So maybe we should rethink our pyramid based on the value provided to the user.
Your users might value speed and ease of use over no failures or downtime. Over a small number of defects. They could value fun over function. Your different user groups might even value completely different things.
So maybe we should have our pyramid set up so that we focus our testing efforts on the things that bring the most value to our users. Does it really matter if there are some defects in the parts that almost no one uses if what our customers come to us for is top-notch?
By looking at things a bit differently, we can optimize the value our testing brings to the organization and one thing is for certain: a happy end user always brings long-term value.
A more agile pyramid
In Agile testing, our goal is not just to reduce the number of defects, but to improve the speed and quality of our iterations. To make the development process as fast as possible, it might make sense to start with the things that save the most time in the long run. Focus your test automation efforts on things that are slower to do. On things that you spend a lot of time testing. If something is fast to do manually, maybe that should not be the first item on your automation priority list.
This type of pyramid would help us focus on things that improve our speed, to improve our velocity. And good velocity helps us bring value to the organization faster and steadier.
You can't protect yourself from unknown risks.
Testing protects you from risks. So maybe we could base our testing efforts on risk analysis. What risks are the worst threats to us as an organization?
Maybe we should spend the most testing effort on those parts where the risks are most likely to get realized. One of the best ways to create value is to prevent catastrophes from happening.
Testing alone does not increase quality.
Maybe we should RETHINK even further
Maybe the pyramid you use should not just be about testing, but about the foundations of quality in your organization.
This pyramid is all about the basis for all of your testing, about what you build your quality on top of. This helps you prioritize things. It helps you see which practices, processes, and artifacts you should definitely own, and which to outsource.
It also helps you communicate better with other stakeholders. What our QA really is all about? Often what others see from our work is just the tip of the pyramid. And it’s so easy to want to look good for others. To avoid focusing only on the most visible parts, it’s good to educate everyone in your organization. Lots of hidden activities, that are not always seen, are crucial for success.
So why did I want to write about pyramids
I’m not saying any of the pyramids I presented here as alternatives are better or worse than the others. In fact, I’m not saying this blog was about pyramids in the first place. This post is all about just making you think. And RETHINK about quality. Never stop rethinking your pyramids.