If you lead a team of developers, SDETs, or QA professionals, you’ve likely had this conversation: should you build test automation yourself, or buy a platform that already does it?
Playwright itself is a strong, well-documented framework, and it has become the default choice for browser automation. That reputation is why the do-it-yourself (DIY) route sounds like the obvious call.
But a free framework doesn’t mean testing is free. Playwright is just the automation engine. Everything that makes it enterprise-ready (authentication, scheduling, approvals, audit logging, reporting, SDLC integration, and more) is work your team builds on top, and that’s where organizations start to lose ROI, often without even realizing it.
We know this firsthand because we built it ourselves. Below is a spotlight on our experience and what we learned.
Week one felt great; week two did not
We wanted to know what it takes to keep a suite trustworthy once real releases depended on it. So, our QA team built our test automation framework using Playwright and Typescript. We used standard practices like Page Object Model and reusable locators, wired into TestRail, with an AI coding assistant to move faster.
Week one was easy: clean framework, core journeys automated, smoke suite running. The first working test scenario took about two hours to build.
Then we connected the scripts to TestRail and checked results against the product. About 80% of our early AI-assisted scripts were technically passing without proving anything: the script finished and reported success, but the business outcome never actually happened.
That’s a false pass, and it’s more concerning than a failing test. A failing test creates work; a false pass creates false confidence, right when you need the real thing.
The cleanup math
Once we knew what to look for, the workload increased quickly. What followed was 50 to 55 scripts requiring manual stabilization because of dynamic IDs, changing text, and timing issues that only surfaced once the suite ran in larger batches.
By month two, we’d rebuilt standalone scripts into a shared, data-driven runner that pulled test data from external sources instead of hardcoding it to cut duplication. This was the point at which the suite quietly became a platform someone had to design, support, and own.
In total, the initial build required three QA engineers working through 284 test cases, for roughly 285 hours, or 36 person-days.
The unplanned hours of review
Building the tests was only part of the effort. Someone still had to verify that a passing test had actually proven what it claimed.
In our exercise, a QA manager needed 20 to 25 minutes to review the expected outcome behind a single test case, and that meant reading the underlying code. Without familiarity with TypeScript or Playwright, the review was not possible, so coverage grew faster than the number of people who could vouch for it.
The unanticipated cost of maintenance
The 36 person-days spent on the initial build were only the beginning; the ongoing cost of keeping the suite reliable is the number that matters, since it recurs monthly.
We modeled ongoing operation using two QA professionals plus supporting engineering and DevOps resources, and the effort broke down into four categories:
- 27 person-days a month on stabilization and locator fixes
- 9 person-days a month on framework ownership and contributor support
- 4 person-days a month on CI, environments, and test data
- 2 person-days a month on failure triage and release review
That adds up to 42 person-days a month, or about 2.1 full-time employees. At a loaded cost of $70,000 per employee, that is roughly $147,000 a year in people costs plus about $25,000 in infrastructure, for a total of approximately $172,000 a year.
The hidden AI bill
One more cost snuck up on us: token consumption. Every generation, debugging, and review cycle pulls in test cases, framework code, page objects, and error logs as context. For roughly 200 test cases, that translates to a directional estimate of 15 to 25 million tokens, which escalates if debugging gets heavy. This is a FinOps line item nobody sees until they go looking for it.
So, build or buy?
None of this means building a DIY Playwright framework is the wrong choice. It is legitimate, provided the organization understands it is also choosing to own a platform: runners, upgrades, access controls, evidence, and the specialists who alone can explain what a passing test actually confirmed.
The real question is not whether a team can build a Playwright framework, but how much of that team’s time should go toward maintaining testing infrastructure instead of writing tests, or building the product customers actually pay for.
Our Playwright experiment, by the numbers
These are the key takeaway figures from our Playwright experiment:
- 80% of early AI-assisted scripts passed technically without proving the business outcome
- 50-55 scripts needed manual stabilization for locator and timing issues
- 284 test cases across 3 QA engineers
- ~285 hours (36 person-days) to build the initial suite
- 20-25 minutes to review one test result, code-reading required
- 42 person-days/month (~2.1 FTE) to keep it running
- ~$172,000/year ongoing ownership cost ($147K people + $25K infrastructure)
- 15-25 million tokens to generate and debug roughly 200 test cases
Try it yourself with Leapwork Play
The enterprise layer around Playwright, authentication, scheduling, approvals, audit logs, and reporting is the gap Leapwork Play closes. And it doesn’t require a team to have to build or maintain it.
In our own comparison:
- A working scenario dropped from two hours to about five minutes.
- Stabilizing after a UI change dropped from 30-40 minutes to about five minutes
- Reviewing the expected result dropped from 20-25 minutes to about five minutes, with no code review required.
The false-pass problem issue we previously encountered didn’t show up in Play’s scenarios. Play uses AI to build and understand tests, then runs them deterministically, with no LLM calls at execution time, keeping your token bill and your trust in the results under control.
Playwright is still a great foundation. The only question worth asking before you scale it is who will owns everything built on top.
Leapwork Play eliminates that question. In its place, Play allows teams to focus on actually using Playwright’s functionality, at an enterprise scale. Try Leapwork Play for free and see for yourself.
