Lucia Cavero-Baptista
Automation is a prerequisite for success with DevOps. But how do you achieve continuous delivery? The answer lies in automation and faster feedback loops.
In one of our previous blog posts, we explained how to build a continuous integration/continuous delivery (CI/CD) pipeline in a DevOps environment. Here we learned that the ultimate success of your delivery pipeline will depend on your automation execution.
In this blog post, we take a closer look at the role of automated testing in continuous delivery.
In the image below, all stages and gates of the release pipeline are color-coded so that it’s easier to identify and understand the flows and checks of the process. Depending on the software and compliance requirements, stages – or environments – might be added or removed from the pipeline, either permanently or by demand.
Furthermore, the gates represent a set of requirements that must be met for a build (a piece of code) to pass to the next stage, or environment, of the pipeline.
The effectiveness of a continuous delivery (CD) release pipeline depends on how fast the results of the Red and Blue Gate checks can provide feedback to relevant stakeholders.
For example, if a unit test fails in Development, the developer who did the check-in should be notified immediately by an email alert.
Or, if the automated UI tests in Test fail, an alert could appear on a dashboard monitored by the team responsible.
The faster the feedback mechanism, the more likely it is to find a quick solution to the given issue.
If a developer is notified weeks after an error is detected in a checked-in piece of code, the context and relevant requirements of the given code are not on top of his or her mind. The developer has already moved on to other tasks and projects, so it will take him time to get his head around it.
The developer would have to spend time readjusting to the right context, resulting in significant productivity loss.
The answer to how to speed up the feedback mechanism in the release pipeline is automation.
Ideally, all the automation needs of the product delivery process should be manageable by a single tool that enhances the work of all the roles involved in the pipeline; from testers to product and business owners.
In the following, we demonstrate how test automation can be implemented in five different parts of the release cycle. We use Leapwork as an example of a test automation platform that works well with DevOps continuous delivery.
1. Regression testing
A primary focus area for automated testing is regression testing. Regression testing helps ensure that changes in product code do not create bugs in the existing product. Regression testing usually takes place in the Test environment.
By relying on Leapwork, and without having to have to write a single line of code, the test team can automate and maintain the test cases required for passing the Blue Gate. The cases can run on a schedule, i.e. every night, repeatedly 24/7, or on an ad hoc basis.
The Leapwork Automation Platform comes with native plugins to the common build and release services such as Jenkins, TFS, Atlassian Bamboo, and TeamCity.
This makes it very easy to trigger test cases to run as part of a release plan. Furthermore, Leapwork returns all test case results in JUnit format, which is natively supported by all platforms. This enables an overview of all test cases directly via any release platform the organization is using.
It is very useful to have individual test cases as a part of more than one schedule. This way, you can test minor parts of the product often while having a schedule containing regression tests of all product features, which is then executed, for example, every night or close to a release.
As mentioned, test automation is meant for making feedback instantly available.
After test execution, all case results are available in the Leapwork Automation Platform, and it is possible to filter them by projects, schedules, and more. The results can also be emailed to designated stakeholders depending on the case outcome.
Finally, test results can be automatically pushed to whichever bug management system is used (TFS, Jira, HP Quality Center, etc.).
2. Early regression/developer verification
Running regression tests against the Development environment serves two purposes:
- Making sure that the build is testable, that is, ensuring that it makes sense to push to Test.
- Providing feedback to developers about whether the Development environment is reliable; if the code works here it can be assumed that it also works in Test and Production.
Early regression testing only covers a small part of the total regression tests available, for two reasons. Firstly, if there is a higher frequency of deployment, or check-ins of code, to Development, there is an upper limit to the number of early regression tests that can be executed per day. And secondly, verification at this stage is usually kept at a limited number of basic test cases that focus on critical features.
The schedules for early regression testing are usually triggered as part of the same processes that generate the build and deploys the code to the Development server. This is easily done with Leapwork’s scheduling capabilities.
3. Feature regression/individual developer verification
The third type of regression testing that is easily automated with Leapwork is feature verification on the individual developer’s local installation.
With Leapwork, teams of testers, developers, and DevOps professionals can share and collaborate on test cases, and everyone can run them on their individual PCs if needed. The results of local runs provide valuable feedback to the entire team.
4. Smoke testing
“Errors will occur.” This is a well-known and accepted truth in software production. So, during a release cycle, it is never a question of ”if errors will occur”, but rather how the organization will react to them.
Enter the discipline of smoke testing.
This is about running automated test cases against the Production environment to ensure that the software indeed runs as intended.
Tests done at this stage are the closest thing to real user interaction. An example of a test case could be to perform a user login to a web application and then validate the login.
The test cases used in smoke testing are solely used for this purpose. The outcomes of the tests are not allowed to dictate any drastic changes to the product. Smoke tests should be able to run both frequently and quickly, and they should not be allowed to push the production system to its limits.
5. Business verification
Whenever the owner of the system under test has to configure, and set up, the system for the end-users, Leapwork can be used to ensure the validity of the system from a business perspective.
The short learning curve of the Leapwork thanks to its flowchart-based approach to automation makes it an ideal tool for business owners, managers, and non-technical specialists.
These people in an organization would be able to design and execute their own automation cases to monitor that the business-critical aspects of the product work, such as a sign-up flow to a subscription service.
READ: How to Automate Functional UI Tests in a DevOps World
Download Whitepaper: DevOps and Test Automation
Do you want to learn more about how to ensure rapid feedback across the CI/CD pipeline through shared automation ownership? Then download the whitepaper: DevOps and Test Automation.
