How to do Regression Testing

Many QA teams feel slowed down by regression testing. But, as a pivotal component of quality assurance and software testing, it’s important to learn how to perform regression testing quickly and efficiently.

In this comprehensive guide, we’ll take you through the steps of how to do regression testing so that you can save time and still enhance the quality of your applications and processes.

Skip to:

What is regression testing?

How to perform regression testing in 8 steps

How to do regression testing manually

How to do automated regression testing

Learn more about performing regression testing

What is regression testing?

The software development life cycle involves planning, developing, releasing, and maintaining software. Part of this process is testing new features before releasing updates to users.

Every now and then, however, updates go wrong.

Despite rigorous testing of new features, existing functionality might be affected unintentionally, causing disruptions to the user experience.

That's why regression testing is crucial.

Regression testing is the process of testing existing functionality when new code alterations are made. Note that regression testing and retesting are two different things.

Regression testing ensures that the new modifications work as intended and that previous functionality remains intact and bug-free. This testing is vital for continuous integration and delivery practices in software development, providing confidence in the stability of your applications with each new release.

How to perform regression testing in 8 steps

Step 1: Identify changes and assess impact

The first step in performing regression testing is to gain a comprehensive understanding of the scope of changes made to the application. This includes new features, bug fixes, and any configuration or environment alterations that might influence the software's behavior. A thorough assessment helps identify the application areas that are most likely to be affected by these changes, guiding the subsequent testing efforts.

For example, if your business is operating on Dynamics 365 ERP, you will need to stay up to date with One Version service releases. These are meticulously documented by Microsoft and shared with customers ahead of the release. Understanding these changes and estimating impacted areas is the first step of your regression testing process.

Step 2: Select and prioritize test cases

The next step involves selecting the tests for your regression suite. You can’t test everything, so determining how to prioritize the test cases is equally important. Use the following prompts as guidance for building your regression testing suite:

• Is the function core to your product? Is it essential for other functionalities to work? Core functions should always be tested.
• Is the feature new, and has it been tested against numerous other feature updates before? New code tends to be more vulnerable.
• Is the code sensitive to the environment being configured? Dependency on environment settings tends to be more vulnerable.
• Has the code been defective before? It’s worth paying extra attention to code that has previously been faulty.

Teams with limited resources or time, especially ones relying on manual testing, may have to take a risk-based approach, where the tests with the highest risk of breakage come first.

Below is a framework with examples that you can use to help organize the prioritization of your regression tests.

Framework for prioritizing tests and building your regression test suite:

Remember that you can always adjust your regression testing suite. As your team develops and tests, you’ll be constantly learning and adapting, so it’s only natural that your test suite will too. It’s crucial, however, if you’re using a regression testing tool, that it provides a good overview and is simple to edit.

Step 3: Develop a regression testing plan

With a clear understanding of the changes and an overview of the prioritized test cases, the next step is to develop a regression testing plan. This plan should detail the testing objectives, scope, schedule, resources, and methodology. A well-structured plan ensures that the testing efforts are focused, efficient, and aligned with the project goals.

To create a realistic plan, you need to be able to estimate the required time.

Use the following checklist to ensure you are considering all the important factors:

✅ Creation of test data: Evaluate the availability of test data, the need for data masking, and whether data cleanup is necessary after testing.

✅ People resources: Assess the number of available personnel, the potential need to involve individuals from other departments, and the team's testing expertise.

✅ Tools: Consider whether you will be automating regression testing. If so, how complex is the automation to build or use, and will you be building it within the sprint? Also, consider whether the automated regression suite operates in the cloud or on-premises and the machine availability for on-premises execution.

✅ Test result review: Estimate the time for reviewing test results. If manual, is it in spreadsheets, and if automated, is reporting integrated, and do you have dashboards visualizing the test results so you can identify and address errors with ease.

✅ Test coverage and complexity: Consider the scope of test coverage and the complexity of the test cases, as extensive and complex tests require more time. You might need to perform some tests manually, so make sure you allocate time for that.

✅ Bug fixing and retesting: Allocate time for bug identification, fixing, retesting, and the cycle that may ensue depending on the number and severity of discovered issues.

✅ Risk management and mitigation: Include time for risk assessment and mitigation strategies, particularly for critical application areas. This might involve stakeholder management, which can also consume unwarranted time.

✅ Quality assurance processes: Are your processes in place, or are you flying the rocket as you build it? The maturity and efficiency of your QA processes can either streamline or extend testing efforts, depending on existing practices.

Step 4: Decide the frequency of your test runs

Once you’ve built your regression testing suite, you need to decide how often you want the regression tests to run.

The picture above paints a clear picture and should serve as a reminder: Every time you let a bug out the door, you’ll want to make sure you haven’t let a swarm of new bugs in.

Regression testing should be performed whenever a change is made to the code. If your software system is large enough, this is only possible to do with automation.

Read How Frequently Should You Run Your Regression Tests to learn how to hit the right regression testing frequency, and to learn about on-demand vs. scheduled automated tests.

Step 5: Leverage test automation where possible

Automation is a key ally in regression testing. Particularly to agile software teams where continuous testing is fundamental.

Get the whitepaper: Continuous Testing In Agile Development

Automation enables faster regression testing and generates more reliable test results than a manual tester chipping through hundreds of test cases.

By automating repetitive and time-consuming test cases, teams can significantly increase their testing efficiency and coverage. Automated tests can be run quickly and frequently, providing immediate feedback on the impact of changes. And the time saved with automation only increases over time, as the same test cases can be reused and rerun without any additional time or effort.

However, it's important to maintain a balance; not all tests are suitable for automation, especially those that require human intuition or are subject to frequent change.

So how do you decide what goes into the regression test suite and what doesn’t?

Below is an overview of tests suitable for automation:

• Highly repetitive tests: Test cases that need to be executed repeatedly over different versions of the software are ideal candidates for automation. Automating these tests saves significant time and effort in the long run.
• Data-driven tests: Automated regression testing is particularly effective for data-driven tests, where the same set of actions need to be validated against multiple sets of data inputs. Automation can efficiently handle variations in data inputs and outputs.
• Load and performance tests: Automated tests are essential for assessing how changes in the software affect its performance and behavior under load. Manual testing cannot accurately simulate thousands of simultaneous users or assess response times effectively.
• Stable areas of the application: Parts of the application that are stable and undergo infrequent changes are good candidates for automation. Once automated, these tests can be run regularly without the need for frequent updates to the test scripts.
• Smoke and sanity tests: Automated smoke tests can quickly verify that the key functionalities are working as expected after a new build is deployed, making them an efficient first line of defense in regression testing.

Types of regression tests suitable for manual testing

• Exploratory testing: This involves testing the application without predefined cases or scripts, allowing testers to explore the software's capabilities and uncover issues based on their understanding and intuition. Exploratory testing is inherently manual and leverages the tester's creativity and experience.
• Tests requiring human judgment: Some aspects of the application, such as user experience (UX), visual appeal, and usability, require human judgment to assess. Manual testing is crucial for evaluating the software from an end-user's perspective.
• Newly developed features: Initially, when features are new and potentially unstable or subject to change, manual testing can be more practical. It allows testers to become familiar with the features and identify any major issues before creating automated tests.
• Complex user interactions: Tests that involve complex user interactions, especially those requiring precise timing or nuanced input, may be more effectively performed manually. Automation scripts can find it challenging to replicate some sophisticated human behaviors accurately.

Step 6: Execute manual and automated regression tests

Run the regression tests as per the prioritization, starting with automated tests for efficiency. Manual testing may still be necessary for complex scenarios or areas not covered by automation.

Step 7: Analyze test results and fix issues

After executing the regression tests, analyze the results to identify any failures or issues. Investigate these failures to determine their cause - whether they are due to new bugs introduced by the changes or other factors. Fix the identified issues, and re-run the tests to verify the fixes. This cycle continues until the application meets the desired quality standards.

Regression testing tools that allow you to easily assess test results and uncover the root cause can mean all the difference for a QA team’s ability to meet release deadlines.

Step 8: Update and maintain the test suite

The dynamic nature of software development necessitates regular updates to the regression test suite. As new features are added and changes are made, the test suite should evolve to reflect these updates.

This includes adding new test cases, removing obsolete ones, and adjusting existing tests to align with the application's current state. A well-maintained test suite is essential for effective regression testing over the application's lifecycle.

Make sure to read our list of regression testing best practices and our guide on how to build maintainable and scalable test automation. Getting informed about the best practices described in these resources will help ensure that you save time on regression testing from the start, and don’t end up building huge, unmanageable regression suites that take up more time than they give back.

How to do manual vs. automated regression testing

The process for doing manual regression testing is slightly different to the process of doing automated regression testing. Following a similar structure to the general process above, we’ll next take you through both approaches, showing how manual and automated regression testing differ:

How to do regression testing manually

1. Understand the changes and impact analysis: Begin by understanding what changes have been made to the application. This could be new features, bug fixes, or updates. Then, conduct an impact analysis to identify the areas of the application that could be affected by these changes. This helps in prioritizing the test cases that need to be executed.
2. Select relevant test cases: Start with your existing test suite. Select test cases that are relevant to the changes made as well as those that cover critical functionalities of the application. Then, modify existing test cases if needed to cover the new changes. Add new test cases to cover any new features or scenarios introduced.
3. Prioritize test cases: Hard prioritization is critical when testing manually - you won’t have time to cover everything. Prioritize test cases based on the criticality of features, the impact of changes, and the risk of defects. Focus first on functionalities that are directly affected by the changes. Then, decide on the depth of regression testing based on the release cycle and the extent of changes. For minor changes, a shallow regression might suffice, whereas major releases might require a deep regression testing effort.
4. Execute test cases: Execute the selected test cases methodically. Follow the test steps carefully and observe the application behavior. Document any deviations from expected results. Capture screenshots, logs, or any other relevant information that can help in debugging.
5. Log defects: Log defects for any discrepancies found during testing. Provide detailed information to help developers understand and fix the issues. Once defects are fixed, verify the corrections in the next regression cycle to ensure that the fixes haven't introduced new issues.
6. Continuous improvement: Use insights and data from regression testing to improve the testing process. Identify frequently failing areas and adjust your testing strategy accordingly. Regularly update your test suite to reflect changes in the application and to remove obsolete test cases.
7. Communication and collaboration: Work closely with developers, business analysts, and other stakeholders. Effective communication ensures that everyone understands the scope of regression testing and the findings. Share regression testing results with the team, highlighting any critical issues found and the overall quality of the release.

An example of manual regression testing

Below is an example of a series of manual test cases that are designed to validate a web shop’s functionality. The requirements and expected results are described on the left side of the spreadsheet, and the results are documented on the right with a simple pass/fail grading system as well as a screenshot.

How to do automated regression testing

1. Select the right tools: The process for automated regression testing starts with the tool, with the assumption that the QA team has a regression testing strategy in place as well. Choose an automation tool that fits the technology stack of your application and meets the team's skill set. Here’s a list of popular regression testing tools and guidance on selecting the right one.
2. Design and develop test cases: Similar to manual testing, identify scenarios that need to be tested based on the changes made to the application and its critical functionalities. Depending on your test automation approach, you’ll need to either develop test scripts by writing code or build them using no-code test automation. Regardless of the approach, make sure the tests are modular, reusable, and easy to maintain. Implement data-driven testing if your test cases need to run with multiple data sets.
3. Maintain test data and environment: Prepare test data needed for automated tests. Use separate data environments to ensure consistency and to avoid corrupting production data. Ensure the test environment closely mirrors the production environment. Automated tests should be run in a stable and controlled setting.
4. Integrate with CI/CD pipeline: Integrate automated regression tests into the CI pipeline to allow for running tests automatically every time code is committed. Tools like Jenkins, CircleCI, or GitHub Actions can facilitate this integration. Ensure that passing the regression tests is a prerequisite for moving code through to deployment stages in the CD process.
5. Execute test scripts: Execute automated test scripts. This can be triggered manually, by a scheduled task, or automatically as part of the CI/CD pipeline. Utilize the capability of the automation tool to run tests in parallel across different environments or browsers to save time.
6. Analyze test results: Automated tools typically generate reports that detail the outcome of each test case. Analyze these results to identify any failures or issues. For any test that fails, log bugs with detailed information for developers to fix. Ensure traceability between test cases and reported bugs.
7. Maintain and update tests: As the application evolves, regularly update and maintain tests to ensure they remain effective and relevant. Maintenance of automated tests can be much more time-consuming, if you’re automating with a script-based approach, than with a no-code approach. Check out our resource on no-code test automation to learn more.
8. Continuous improvement: Use insights from automated testing to refine and enhance your testing strategy. Identify areas for increased coverage or where manual testing may still be necessary. Continuously optimize the test suite for performance, removing outdated tests and adding new ones as needed.

An example of automated regression testing

Automated regression testing varies depending on the size and complexity of the software and the tool used. Below is a comparison of two automated regression test processes. One is code-based, and the other is no-code and completely visual in setup.

Read these case studies to see real-world examples of how businesses have introduced automated regression testing:

➡️ Specialist bank, Investec, automated 95% of their regression suite to reduce risk across the organization and deliver 3x faster time to market.

➡️ JF Hillebrand, a logistics company, fully automated its regression testing of web apps in three months. Through automation, they reduced testing time from 3 days to 40 minutes, a 97% reduction.

Learn more about performing regression testing

Continue learning about automated regression testing in our webinar on automated regression testing. Or, check out our whitepaper on the topic to learn more about regression testing in agile teams.