The following post is a first-person, eyewitness account from Leapwork CTO Rohit Raghuvansi about how adopting Leapwork Play into his QA team’s workflows enabled them to increase new feature deployments from every two weeks to on-demand. Below, Rohit reflects on how aspirations for improving deployment speed led his team to discover firsthand how using Leapwork Play to manage their regression suite environment increased predictability to the degree of allowing them to release more confidently at a faster rate.
There was a time when I was leading QA, and my team was deploying new features once every two weeks. That frequency was not enough for us; we wanted to deploy on demand.
One thing that appeared to prevent this was our deployment frequency. Whenever our deployment frequency slowed down, we first looked at CI/CD.
CI/CD seemed like the obvious place to start. We reviewed build times, pipeline stages, deployment scripts, and infrastructure capacity. We discussed whether more steps could run in parallel and whether additional computing power would make the process faster.
I initially agreed with that diagnosis. If we were not releasing frequently enough, the deployment process must be taking too long.
But the numbers did not support it.
Our builds were completing within a reasonable time, and the deployment pipeline was ready to release. The delay usually started after the build was available, when my team began regression testing.
A release would appear ready from an engineering perspective, but we could not confidently approve it. The business would ask why we could not deploy that evening. Development would confirm that the build had passed. My team would still be investigating regression results or preparing to run part of the suite again.
The problems at hand
From the outside, it looked like QA was slowing down the release. But when I looked more closely, I realized the regression tests themselves were not the main problem.
The real problem was the environment in which they were running.
On a good day, the suite completed within the expected time and produced a clear result. On a less-than-good day, the same tests took much longer or failed without any meaningful change in the application.
We saw the following:
- A service might respond slowly
- A machine might not be in the expected state
- Test data might need to be reset
- Browser sessions from an earlier run might still be active
- One failed execution could leave the environment in a condition that affected everything that followed
This made every failure difficult to trust.
Investigating the issues
When a test failed, my team could not immediately investigate the product. We first had to determine whether the environment had caused the problem.
We asked ourselves questions like:
- Was the required service available?
- Was the machine overloaded?
- Was the test account still valid?
- Had an earlier test left the application in the wrong state?
Sometimes we would restart the environment, rerun the same test and see it pass without changing the product or the test. That meant the first result had told us very little.
Before an important regression run, someone from my team would check the machines, validate accounts, clean up old sessions, reset data and confirm that all dependent services were working. If several tests failed together, we would stop, repair the environment and run them again.
This work was mostly invisible to the rest of the organization. People could see that regression was taking too long, but they could not see how much effort went into preparing the environment and separating genuine product failures from problems with the setup.
My engineers were busy, but too much of their time was being spent maintaining the conditions required to test rather than testing the product itself.
We tried to improve the process manually. We created environment-readiness checklists, added cleanup scripts and introduced health checks. We documented the services that needed to be verified and assigned people to monitor important regression runs.
These changes helped, but they did not remove the dependency on people. Someone still had to notice when the environment was unhealthy, understand the problem and fix it before testing could continue.
Growing pains
As the product grew, the regression suite grew with it. That meant more machines, more accounts, more services, more test data and more opportunities for the environment to fail.
We had automated the tests, but we had not automated enough of the work around the tests. That was the real reason we could not deploy on demand.
Our deployment pipeline was capable of releasing quickly. What we lacked was a dependable way to complete regression testing and give the organisation an answer it could trust.
This changed when we started using Play.
How Play changed the paradigm
We were not looking for another tool that could simply execute automated tests. We already had those tests. What we needed was a better way to manage the environment in which they ran.
Leapwork Play took care of much of that work for us. Instead of my team repeatedly preparing machines and repairing the setup, Play provided the required execution environment for the test run. The tests ran under more consistent conditions, and the environment did not carry the same problems from one execution into the next.
The immediate benefit was predictability.
My team spent less time restarting machines, cleaning old sessions and resetting the setup. Test runs became more consistent, and we needed fewer reruns.
More importantly, failures became easier to trust. When a test failed, the team could focus on what had changed in the application rather than first proving that the environment had behaved correctly.
This changed the quality of our release discussions. Earlier, the question after a regression run was often, “Can we trust these results?” Now the discussion was closer to, “What does this failure mean for the release?” That is a much more useful conversation for a QA team to have.
The improvement in deployment frequency did not come from a major redesign of our CI/CD pipeline. It came from removing the uncertainty between a completed build and a confident release decision.
Once regression became predictable, we could test faster, understand failures sooner and approve releases without waiting for repeated executions. That is ultimately what made deployment on demand possible for my team and I.
Reflections on Releases
This experience completely changed the way I think about release speed.
A company may have an efficient build system and a fully automated deployment pipeline, but it will still struggle to release frequently if regression testing depends on an environment that needs constant manual attention.
The speed of the deployment pipeline is only one part of deployment frequency. The other part is how quickly QA can give the team a result that everyone trusts.
For us, the bottleneck was not CI/CD. Instead, it was the environment quietly sitting behind our regression suite.
Once Play started taking care of that environment, my team could focus on the product and the actual release risk. Then, we could finally deploy when we were ready.
