Testing in SAP: An Introduction to Automation Approaches

Jump to section:

Watch Webinar
Book a meeting

With regular updates and customizations in a business's SAP system, testing is a crucial part of the development and maintenance of SAP.

Many perform testing manually but when it is done manually, it takes longer, and it puts the security of business-critical SAP processes at risk.

To solve this challenge, businesses are adopting automated testing. Not all test automation solutions are easy to use though. Code-based solutions, while effective for some, rely on expensive developer resources to build and maintain. This adds a layer of complexity to an already complex system.

So what are the options for businesses looking to accelerate their testing, without putting the quality of the SAP system at risk?

In this extended article, you’ll learn about what you should be testing in SAP, the pitfalls to avoid when using automation, and the benefits of maintainable and scalable automation.

What is SAP?

SAP is an enterprise software used by major companies for customer relations and business operations. The mammoth platform is made up of six core products, each with several sub-products:

  • ERP and Finance
  • CRM and Customer Experience
  • Network and Spend Management
  • Supply Chain Management
  • HR and People Engagement
  • Business Technology Platform

Combined, these core products provide enterprises with an IT system for managing business processes, within financial services, transport, energy, and other sectors.

SAP is often customized to suit a business's unique needs. Often, SAP integrates to other non-SAP systems as well. For example, there may be an integration between an Enterprise Resource Planning (ERP) system like SAP and a Customer Relationship Management (CRM) system like Salesforce.

Because of these configurations, the likelihood that something will break is high. To make sure that business processes within SAP and their integrations with external systems work, testing is carried out continuously to make sure SAP is secure and bug-free.

1. Why should you test SAP?

If a bug goes undetected in an SAP module, the consequences can be disastrous.

One example of a worst-case scenario coming to life is the launch of SAP for the National Grid. Back in 2012, the American-based utility company was due to launch its SAP implementation.

The pressure to release was high as it was overdue. The cost of missing their go-live date would reach the millions, so there was no option to delay. Then Hurricane Sandy hit and obliterated much of their service area.

Even so, they flipped the switch on the go-live date, and chaos ensued. Paychecks were incorrect, with some employees being underpaid, others being overpaid. Vendor invoices weren’t being processed, and their financial reporting didn’t work. This meant that they were unable to get short-term loans they were reliant on for cash flow.

What this example goes to show is that SAP is an incredibly complex system used to run many important business processes. Because it drives so many processes, the cost of a bug can be astronomically high.

Averare cost of a software bug

When your company is reliant on SAP to run vital business processes, testing customizations, updates, and integrations with other systems is crucial to the continued functionality of SAP.

2. What to test in SAP

There are many instances that make SAP systems vulnerable to errors. In summary, these are:

What-to-test-SAP

  • Customizations. In SAP, customizations are termed WRICEF’s. WRICEF is an SAP acronym covering any kind of custom development or enhancement. It stands for Workflow, Report, Interface, Conversion, Enhancement, and Forms. As the default features don’t always support the features a business needs, customization is common. These customizations must be tested to make sure that they have not caused an error.
  • Updates/Configurations. Whether your business is working in SAP S/4HANA where updates are more regular, or in an SAP module, updates will take place. Depending on the company, the frequency of these updates will differ. However, when there is a system update, it can cause a business process to break. To make sure this is found before anything is released to a live environment, testing will make sure that a bug is found and fixed. You can learn about the difference between customizations and configurations below
  • Integrations. As we established in the introduction, every module in SAP supports a business process. A business process can run across more than one module, but it can also extend beyond SAP to another system such as a desktop application (e.g., Excel/outlook), a legacy system like Mainframe, and/or a web application like Salesforce. Just as customizations and configurations are tested, so are integrations. You can find examples of integration tests below.
  • Transitioning to SAP S/4HANA. With a deadline for 2027, SAP ECC is transitioning to S/4HANA for easier use and the capability to handle more data. Businesses will need to test their solution, processes, and integrations before undergoing this transition. Some of these tests include doing end-to-end automation in and outside of SAP, generating master data, and build flows to verify processes to make sure the system is ready for the move.

Preparing for an SAP S/4HANA transition

With a deadline for 2027, businesses that are undertaking the migration to SAP S/4HANA need to have an action plan in place to ensure a smooth transition.

A roadmap for the migration can be created based on data from the SAP Readiness Check 2.0 Tool (SRC2.0). This tool gives an “overview of the compatibility of the current system with the updated applications in the target release (for example, upgrading to SAP S/4HANA 2020) and the preparation steps that will be required.”

Along with SRC2.0, SAP’s own set of tools help you through the transition journey. These include the Maintenance Planner, the Custom Code Worklist, the Software Update Manager, and the Value Assurance Packages. They also offer a roadmap for mapping out the transition process from beginning to end (based on the input provided).

However, testing is still a fundamental step in the success of a transition. Some solutions, processes, and integrations in SAP will need to be tested before others, or they’ll be contingent upon one another.

Employ a cyclical approach with testing gates to check processes are running as intended before moving onto the next stage. By setting up testing gates you will be able to catch issues and fix them before building on top of a broken build.

For this reason, test automation is a crucial component in your transition. Automating the testing of your system will lower the risk involved in transitioning to S/4HANA.

Read more about S/4HANA migration readiness in this extended guide, written in partnership with SAP experts at NNIT.

The difference between configuration and customization in SAP

  • Configuration. Using a native tool to change a feature or behavior.
  • Customization. A modification, feature, or extension is created through a special implementation or custom coding.

Examples of common integration tests in SAP

  • TPP order system integrations. Making sure that new purchase orders are immediately made available and transmitted to SAP.
  • CRM integrations. Companies need to sync data between their ERP (e.g., SAP) and CRM (e.g., Salesforce) systems. If, for example, a new customer is onboarded, the CRM data must sync with financial modules, performance management and potentially other business functions within SAP.
  • Supplier system integrations. If a purchase requisition is created for a raw material or component in SAP, once the purchase has been approved it needs to be sent to suppliers. These suppliers are inevitably using separate systems when they respond to the request with a quote. These quotes are routed back to SAP for processing. To make the process more efficient, SAP would likely integrate with the supplier systems.

3. Types of testing in SAP modules

As we established in the previous section, upgrades, customizations, and integrations mean that tests must be carried out in SAP on an ongoing basis.

Types-of-testing

There are several types of tests that SAP testers must perform:

 

  • Unit testing. In short, unit testing is testing a specific functionality in silo. It would take place when a WRICEF is complete, or a configuration is complete. For example, if you are working on a form, you would carry out a unit test on each item of the form. Unit testing makes sure that the configurations or customizations are working as they should. It is typically done once a “unit” is complete. The developer who created the configuration will often be the one to test it, otherwise, it is done by a consultant hired for the purpose. This takes place in a development system.
  • Integration testing. Once all units are complete, you would typically move on to integration testing. In this stage, you test all the different pieces together. It’s an end-to-end test walking through business processes from A to B to make sure they work. In this stage, you could be testing an end-to-end process across different SAP modules, and across non-SAP systems. This is typically carried out in more than one cycle by a dedicated test team, a consultant, and a couple of key business users. Unlike unit testing, integration testing is carried out in a quality system and test system (in some cases). To make integration testing more efficient, this stage can be automated.
  • Performance testing. Performance testing makes sure that functional parts of the system are responsive when in use. This could happen on an input form or document on SAP. Performance testing is also called stress testing and it is often automated. It can be done in conjunction with integration testing, but it can also be done after integration testing. It is normally carried out by an IT team, or a dedicated performance testing system.
  • Functional testing. Functional testing consists of reviewing design documents and creating test artifacts including test requirements, test scenarios, and test cases. Functional testing is usually done by a testing team with a background in the SAP module being tested. Many businesses choose to automate functional testing to speed up the testing process.
  • User acceptance testing. At this stage, the system will be in a mature state, most bugs will be resolved, and end-users will be testing. Testers either follow a test script, or they perform exploratory testing. Like integration testing, it will also take place in a quality system, and it is one of the last phases before going live.
  • Regression testing. Once everything is in production and users are active, an error might be found. Or a user could request a new feature. Here, testers make sure that everything in production, and any new features that are released after the live date, work. Regression testing makes sure that whenever a change is made, the whole system is tested to verify that the change has not caused issues in another part of the system. It can be done manually, however, most businesses automate regression testing.

For the remainder of this article, we will be referring to migration, integration, and regression testing most frequently. It is within these stages that most development projects are held up. There are many reasons for this.

However, two challenges we see occurring frequently are related to manual testing (time constraints), and automation testing (specifically the problems that come with scripting and maintaining automation). The problem with manual and automated testing in SAP will be explained in more detail below.

4. The challenges that come with manually testing SAP

SAP is a vast system and can cover most major and minor processes in a business, from invoicing to recruitment, and supply chain management. Even the tiniest error can cause wide-scale failure across the system. Naturally, testing has become a fundamental part of the development process.

With configuration and customizations taking place frequently, SAP and its integrations must be regularly regression tested to verify that the system is not negatively impacted by a change.

To do this process manually is incredibly challenging. But for many, it is the only option.

Not all enterprises have available development resources to build automated tests. Naturally, the responsibility is divided between those who are experts in the system – business users.

In the likely situation that you are assigned manual tests, it can take several days. All the while, business-essential work is being ignored. This slows down the release of customizations and configurations to a live environment.

In sum, when testing is performed manually, it takes longer, and there is a higher risk of error. For enterprises with large and complex digital ecosystems, relying completely on manual testing is not an option.

Below you’ll find an overview of the automated testing solutions, from code-based solutions to low-code and codeless solutions.

5. Why you should automate testing in SAP

Before starting this section, it is important to note that it is a challenge for SAP to implement and agree on the best practices and standards for SAP test automation.

There are solutions available that can make SAP testing faster, easier, and with less maintenance. But a one-size-fits-all solution for testing does not exist. Every SAP environment and organizational structure will be unique, hence the right solution for one business may not be the same for another.

Testing SAP with the help of automation is the natural solution to make the development process more efficient. It doesn’t require days of manual testing, and regression testing can be run with the click of a button.

Related reading: How Can SAP Be Automated?

The downside is that many automation vendors offer highly technical solutions that require expensive developer resources to use and maintain. Because SAP is built upon a proprietary programming language (SAP ABAP), it is harder to find a suitable tool outside of SAP’s own offerings (e.g., SAP TAO and eCATT).

With SAP TAO coming to an end, this limits the selection further.

Below you can find a summary of the most common (code-based) solutions used by businesses today, along with their advantages and disadvantages.

6. The code-based test automation tools for SAP

SAP automation testing using Selenium

Selenium is a popular testing tool used by testers with technical skills to automate testing processes like functional testing and regression testing.

It can be used across browsers and operating systems, and it works with several programming languages making it ideal for companies with complex software spanning across web browsers.

The benefits of Selenium for SAP

  • It’s free, as opposed to most other automation tools that include licensing costs
  • It’s open-source, allowing extension and modification of source code
  • It supports multiple programming languages
  • It supports most operating systems
  • It supports all major browsers
  • It has community support, due to the large network of users
  • It has integration options, allowing e.g., parallel testing and reporting

The drawbacks of Selenium for SAP

  • It’s difficult to set up and use, potentially leading to high start-up costs
  • It’s difficult to maintain
  • Test development is slow as all test cases need to be scripted
  • It only supports web browsers, not mobile, desktop, virtual or desktop
  • Integration with the rest of the pipeline is difficult
  • There is no dedicated support, only user communities
  • It requires third party solutions to cover all testing needs

Extended Computer Aided Test Tool (eCATT)

eCatt is SAP’s in-built functional testing and integration testing tool and offers an ABAP script editor. Users create and execute automated tests of business processes in SAP. A common use for eCATT is automating regression testing if the system is extended or a support package is installed.

The benefits of eCATT

  • It’s free of charge so you don’t need to acquire extra licenses
  • It’s managed and maintained in one central system
  • You can access all layers of an SAP system (database, application layer, and presentation layer)
  • A modular approach to testing means users can reuse test and system data
  • It can integrate external tools for extending reach to non-SAP systems

The drawbacks of eCATT

  • Limited to SAP UI and API technologies. This means it only works in an ABAP development environment. HTML, for example, is not supported.
  • Non-technical testers and business users can’t use this tool as you must know how to program
  • The upkeep of the test script requires a lot of maintenance work

Quick Test Professional (QTP)

QTP is a keyword-driven automation tool for functional testing in web and desktop applications. It is based on the VB scripting language. Organizations often chose to go with QTP rather than eCATT as it supports technologies other than SAP GUI.

The benefits of QTP

  • Record and playback features for quick test case design
  • Comprehensive training material available
  • Built-in REST API

The drawbacks of QTP

  • Without a structured approach, the maintenance of QTP scripts can render automation code unusable
  • Script-based testing prevents non-technical testers from automating complex tests
  • Adoption time can take between 6-9 months
  • Programming is required for cross-technology support

SAP TAO (Test Acceleration and Optimization)

SAP TAO is used as an aid for building automated tests in SAP. It is primarily used for end-to-end tests because it helps to break down software into smaller components, making automation more manageable. However, the tool is being phased out, so businesses using TAO will need to search for an alternative.

You can learn about the benefits and drawbacks of SAP TAO in this comparison chart, where we compare different testing solutions for SAP.

Test Automation Tools for SAP

An overview of what to look for in an SAP test automation tool and the alternatives to SAP TAO.

7. The challenges of code-based automated testing in SAP

Restrictions of SAP’s own testing tools

As we established in the previous section, SAP’s own testing tools are limited in terms of what you can test. For example, if you are using eCATT, you can access all layers of an SAP system, but you will not be able to automate beyond SAP UI and API technologies.

test-automation-limitations

Scripting automation

Most tools for test automation offer a function where you can freestyle programming. Using this function does not always make sense. It can increase the maintenance work required to keep the automation running, and it can make it harder to reuse tests.

For example, if incorrect data types are used and naming conventions for scripts and parameters are not followed, it will be difficult for others to reuse the script.

Reusing test automation script

As those working in big organizations will know, trying to get people to reuse the same code is a never-ending struggle.

New people come on board and will have their own preferences for scripting. Or a user may struggle to find reusable modules. Rather than spending the time identifying the reusable script, it can seem easier and faster to create your own script to meet your own expectations.

The skills gap between testers and developers

In most testing teams, the roles can be divided into two. The tester, and the automation technician.

The tester, in short, takes care of the definition of what is to be tested, the creation of test scripts (not to be confused with scripting test automation), along with the quality of the tests and the effectiveness of the testing.

The automation technician, in short, takes care of the efficiency of the testing and creates the automated processes.

In some cases, this role is assumed by one person. In others, it can be divided between several.

When you couple this with the fact that the skills required from professionals are changing, problems start to arise. Businesses are looking for profiles with experience in development and testing. A team with multi-skilled personnel is attractive for obvious reasons – your team will be able to deliver tasks faster, and more cost-effectively. However, not many of these multi-skilled professionals are available. And those that are available are a costly resource.

Adopting tools that enable non-technical users to automate therefore closes the skills gap that will inevitably arise in most test automation projects.

Related reading: How No-code Test Automation Closes the Skills Gap in Software Development

Resisting test automation

Typically, you will experience pushback from non-technical testers (i.e., functional testers, business analysts, etc.) when introducing test automation. This reaction can (in some cases) whittle down to two reasons:

  1. They fear it will take over their job
  2. They have a very heavy workload and do not see the value in creating automated tests if it will not immediately relieve their workload

If you’re facing the first dilemma, it is important to reassure your team that test automation is not here to replace jobs. It will take away the boring tasks like regression testing, so they can focus on the important and challenging tests.

Now for the second dilemma. If your testers are very busy testing the system, they will feel as though there is no time to take on additional projects. If the automation tool takes too long to learn because it is script-based, it is likely that your team will not adopt it. This will make it very difficult to scale automation beyond siloed projects. Forcefully “spoon-feeding” the solution will not make the problem go away.

The path to least resistance

In this case, it is important to adopt an automation solution that has a short onboarding time and is easy to use for business users and testers. Code-based solutions make this difficult (see reasons above), while visual solutions that don’t require coding experience make it easier and faster for the entire team to build, reuse and maintain tests.

8. What you should look for in an SAP test automation solution

As we covered in the “Why you should automate testing in SAP” section, not all test environments are equal, and the requirements will differ greatly from one company to the next.

That being said, there are common threads that appear across most organizations in terms of the basic functions a test automation solution for SAP should have. The following are listed below:

  • A short learning curve, enabling testers to set up test cases from day one
  • Easy-to-use, enabling anyone to set up, maintain and run tests
  • Scalable, allowing you to run your entire regression suite at high speed and frequency

Related reading: How to Increase Your SAP Regression Testing Coverage

  • Cross-technology automation is possible, working across all web, mobile, and desktop apps and technologies
  • Keeps a record, capturing why test cases fail in videos and logs
  • Reliable, keeping your data and most business-critical processes safe
  • Supported, getting you help when you need it

 

Guide to Automating SAP

Learn how to get started with SAP test automation and RPA fast and efficiently for rapid value creation. 

9. Automating SAP with a codeless solution

By adopting codeless automation, you can increase the quality of your software and in the process, better prepare for updates and changes to your SAP system. Whether it’s for specific use cases, like planning tables used in SAP APO, or to serve as a replacement for SAP’s own testing tools.

With a tool like Leapwork, you aren’t dependent on programming knowledge of SAP’s proprietary language – ABAP. Instead, you can tap into the potential of those testing your system – the business users.

Learn more about Leapwork for Testing in SAP

sap flow

Listed below are some of the capabilities of Leapwork for SAP:

  • It’s easy to use and adopt. Business users and QA managers can take ownership of SAP automation. With the fastest learning curve on the market, testers are equipped to automate in less than three months.
  • Mitigate risk at scale. Quickly ramp up automation coverage to ensure end-to-end business process continuity across your organization.
  • Say goodbye to heavy maintenance. With hyper-visual debugging tools, maximum reusability with reusable components, and data-driven capabilities, maintenance is extremely efficient, no matter how complex your SAP system requirements. No coding skills needed.
  • You can release with quality SAP customizations and implementations at speed. Accelerate the delivery of SAP updates, modifications, and service pack rollouts and release with confidence.
  • Users can collaborate seamlessly. Enable people with different skills and expertise to work efficiently together and automate SAP.
  • Migrations to S/4HANA. You can protect against downtime with Leapwork’s rapid automation of your testing. You’ll achieve maximum coverage of your system so that you can migrate your system with confidence. With a network of consulting partners, you can deliver automation faster. Read our solution brief on Leapwork for S/4HANA migrations.
  • Automate across all technologies. Whether your application integrates with multiple web-based portals, or you need to run data-driven tests for third-party customer success tools, we’ve got you covered. With Leapwork, you can use the same visual approach to build automation across all your integrated applications.

cross-technology-circle-8

You can learn more about automating SAP securely and at speed in this solution brief, or watch a demonstration of codeless automation in action in SAP in our on-demand webinar.

leapwork-resource-webinar

Watch the on-demand webinar on automating SAP with Leapwork

Learn how to speed up the testing process for your SAP instance and mitigate risks caused by customizations or system updates.