Aske Denning
To keep up with the ever-increasing customer expectations for digital products and services, software providers are under pressure to be more and more thorough in their quality assurance of products, and to deliver that quality at speed. Read on to learn how continuous testing can support those efforts.
An effective way to improve quality assurance (QA) is to fail fast.
The purpose of failing fast is to catch and fix a bug as close to its origin as possible, before it makes its way further down the release pipeline.
Because it’s much easier and less costly to fix an error in the context of which it was created, it’s always preferable for an error not to change hands.
The traditional waterfall approach to software development does not support the objective of failing fast, as software testing is postponed until the final stages of a release. As such, another approach to QA is required.
Enter continuous testing.
What is continuous testing?
The move towards a more agile production of software has introduced the concept of Continuous Delivery (CD).
Author and long-time developer Jez Humble defines CD as “the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.”
A common challenge when developing software following CD principles is that you can only deliver as fast as your slowest asset.
After a feature has been developed, but before it is released, what's most time-consuming is testing and all its related activities. Testing can be sped up with automation, and at the same time increase accuracy and thereby lower risk, which overall enhances product quality. This is why automation is key in agile.
Whereas agile development does not in itself ensure good quality, continuous delivery aims to help answer the question “What’s the quality of our software right now?” at any given point in the pipeline.
Continuous testing is a means to be able to answer that question, as this method is all about:
- Testing at earlier stages of the release pipeline
- Testing more often before release
- Testing everywhere, that is, across environments and devices
Continuous testing helps software teams approach the ideal of being able to evaluate quality at several stages of a pipeline where code is continuously developed, delivered, and deployed.
Why the waterfall approach doesn’t work anymore
Testing continuously is about testing as soon as a piece of code is written, as opposed to testing it at the very end. This is also the idea behind CI/CD.
This comes as a reaction to the more traditional waterfall approach, where product development would be siloed, and individuals on the team would work separately, passing on their piece of the puzzle when it was complete on their end.
The problem with the waterfall approach is that it could take days or weeks for a piece of newly written code to be committed, tested, and validated. As a result, developers risked wasting needless time on writing on top of broken code, and it would be difficult to identify the source of a bug, as well as to fix it, the longer it took for a bug to be found.
In continuous integration, continuous development, and continuous testing, code is committed by developers and merged with other developers’ every day, sometimes several times a day, and it is then tested right away.
As a result, bugs are caught earlier in the cycle.
This is all made possible by automation - not just in testing, but throughout the pipeline.
The role of test automation in continuous testing
In practice, continuous testing is achieved by automating tests (e.g. regression, functional, integration, and performance tests).
Automation allows the tests to be executed as soon as a piece of code is ready at their respective stages of the release pipeline. Because it doesn’t require human involvement, dependencies and bottlenecks are removed, speed is increased, costs are lowered, and the risk of human error stemming from repetitive work is reduced.
To adopt continuous testing successfully in a CI/CD setup requires a test automation solution that’s a good fit for your pipeline.
It’s important to note, however, that test automation does not in itself enable continuous testing, but it is absolutely a prerequisite for scaling testing – whether continuous or traditional.
Finding and implementing a test automation tool can in and of itself be a challenge. With the wrong tool and approach, there’s a high risk that teams won’t see the desired benefits of automation.
Common challenges in continuous testing
Building a CI/CD pipeline and achieving continuous testing with automation can be a challenge. Some of the most common barriers to success are figuring out what to automate, how much to automate, and when to automate it.
It can in general be a good idea to adopt a test automation strategy, rather than automating tests in an ad-hoc manner.
In addition, creating test environments and configuring an automation tool can also be a major challenge and is something that, in many cases, require heavy start-up costs in terms of developers’ time and resources.
The biggest hurdle of all is for most QA teams in maintaining test automation. Oftentimes, developers become bottlenecks because they are the only ones who know how to build and fix the automated tests.
No-code test automation can help teams overcome this challenge, as the learning curve is significantly shorter, maintenance can be done by developers and testers, and collaboration is enhanced as anyone on the team can understand the automated tests.
Related reading: No-code test automation and how it contributes to seamless, continuous testing
Continuous testing tools: What to look for
Finding the right test automation tool is a crucial part of enabling continuous testing.
Since speed is a key, a tool that will let teams work fast and easily together is a must. As mentioned above, no-code tools will let teams move faster than code-based solutions will.
Benefits of continuous testing
With the right tool and approach, teams using test automation to achieve continuous testing can expect to see the following positive outcomes:
- Faster releases: With faster and more continuous feedback loops throughout the pipeline, bugs can be found earlier and fixed earlier. This also means quality can be assured faster, and releases can take place sooner.
- Higher job satisfaction: By letting humans do human work and robots do robotic work, resources are spent better. Developers can focus on developing, and testers can ditch repetitive, tedious tasks in favor of tasks that require their critical, creative thinking, such as exploratory testing. As a result, test coverage and quality can be increased.
- Lower risk: Robots don’t get tired or bored, and are therefore much better at performing highly repetitive tasks with large quantities of data. They can perform these tasks faster and with higher accuracy than humans, and can therefore be used to effectively mitigate risk in this area of testing.
- Lower costs: When testing is performed faster and with higher precision, and resources are optimized, costs are optimized or reduced as a natural result. Furthermore, when bugs are discovered earlier in the release cycle, they are less costly to fix than later.
- Higher quality: Faster testing cycles doesn’t mean that testing is rushed - rather, it means that the depth and breadth of testing is increased, that more testing can take place, and higher product quality is therefore a natural result of continuous testing.
Conclusion
Continuous testing is about thorough, frequent, and reiterative execution of tests along the release pipeline. It is a part of CI/CD, where automation is a prerequisite.
Continuous testing enables early discovery of errors, and combined with other initiatives, such as Scrum, continuous testing helps you achieve agile development. Some of the benefits of testing continuously include reduced risk, lower costs, and higher product quality.
To achieve continuous testing requires the right tool and approach. We have many resources where you can learn more about continuous testing in related contexts, such as agile and DevOps.
Download our whitepaper on test automation and DevOps to learn how to implement test automation in the CI/CD pipeline quickly and easily.
