If you ask a roomful of QA engineers how to fix a flaky test, you’ll receive no shortage of answers.
“Add retries, improve synchronization and stabilize test data,” some will tell you. “Isolate dependences and update selectors,” others will say.
This is valid advice. These practices will indeed help reduce testing flakiness.
But the problem facing the typical organization today is across-the-board flakiness. At that level of scale, the underlying issue is not missed optimization opportunities in individual tests; it’s flaws in the organization’s core approach to tracking and managing flaky tests.
The scope creep of flaky tests
When flaky tests emerge, organizations tend to respond in an ad hoc fashion, attempting to deal with the tests one by one. They do things like export test results to spreadsheets, build dashboards on top of time-series databases, and leverage CI logs to identify recurring failures. They also tend to normalize a reliance on institutional knowledge by assuming that experienced engineers will simply know which test failures they can ignore.
This approach works well enough for small test suites. But the more tests you have to manage, the less effective and practical it becomes, due to issues like:
- Limited context: When you track tests individually into gaining holistic visibility into the status of all of your tests, you deprive yourself of the ability to identify overarching trends, such as which types of tests correlate with the highest levels of flakiness.
- Inability to detect product regressions: Lack of context also means that teams struggle to distinguish between a genuinely unstable test and a legitimate product regressions.
- Poor test confidence: Over time, these deficiencies undercut the organization’s confidence in test results, which causes frustration and morale loss among engineers, while simultaneously increasing the risk that the business will release deficient products.
Scaling flaky test management
Conquering test flakiness at scale requires teams to think more holistically and strategically, and part of the solution involves adopting dedicated tooling. Instead of ad hoc measures like spreadsheets, engineers should adopt a platform that allows them to track tests and assess trends comprehensively.
Enforcing ownership over tests is another critical component. Teams need a structured framework for designating who is responsible for uncovering the root cause of flaky tests, then implementing and validating fixes. Strong governance, is also essential: Governance allows teams to define clear, consistent policies for assessing test results, quarantining tests and so on.
Put together, investments like these empower the organization not just to manage test flakiness more efficiently, but also to gain full confidence in test results. This, in turn makes engineers happier while reducing risk for the business.
This is precisely the type of win-win scenario that the Leapwork Continuous Validation Platform was built to deliver. When users build tests in Leapwork Play and Leapwork Flow, self-healing automation adapts as interfaces and systems change, reducing the flaky test triage cycles that consume engineering capacity month after month. Users can also self-heal tests automatically when an element changes, allowing QA to repair a native-Playwright test without requiring an engineer to intervene.
