You point an AI at your app. You give it a prompt about as detailed as “explore this site and write tests.” You go make a coffee. You come back to 200 Playwright tests, all written, all runnable, and (this is the really enticing part) all green. It genuinely feels like magic. A week of work, done before the coffee got cold.
Then a slightly uncomfortable question shows up, usually from the one engineer on the team who’s been burned before: “Okay, but how many of those actually test anything?”
It’s a far more realistic question than a cynical one, because a test can pass and read perfectly well in review… yet never once exercise the behavior it claims to cover. Teams are hitting this at scale right now, and the community has already coined an unflattering name for the output: AI slop tests.
Below, we discuss how you could end up with a suite full of them without noticing, and what it takes to trust generated coverage when a release is on the line.
Green doesn’t mean good
When the demo goes well, nobody tells you that a passing test and a useful test are two wholly different things that happen to look identical on a dashboard. Both show up green, but only one of them is actually standing guard.
Picture a “test” for your checkout flow. It navigates to the page, checks that a button labeled “Pay now” exists, and clicks it. It confirms the URL changed; that it shows up green, and then it ships. Except it never verified that a payment was taken, that the right amount was charged, that the cart contents matched the order, or that a failed card showed an error instead of silently swallowing the money.
Instead, it tested that a button is a button. Congratulations.
That’s the trap. The most common failure mode isn’t a test that breaks; it’s a test that passes while checking almost nothing. These tests never fail loudly. They just quietly hand you confidence you never earned.
A silent false pass is more dangerous than an outright error, because an error at least tells you something’s wrong. A hollow green test tells you everything’s fine, right up until it isn’t, in production, on a Friday.
Why AI loves the happy path
A test that passes but checks nothing is the model doing exactly what you’d do if you were dropped into an unfamiliar app with no context and told to “write some tests.” You’d click around, see what works, and write down what you saw working. That’s the happy path: the clean, sunny route where the user does everything right and everything behaves.
Trouble is, that’s not where bugs live. Bugs live in the ugly corners: the expired session, the double-click, the negative quantity, the special character in the name field, the network blip halfway through a submit. Those are the cases a human tester learns to be suspicious of. An AI exploring your UI has no reason to suspect them, because nothing on the screen advertises them. So, it writes forty tests for the front door and none for the windows.
It’s worth being clear-eyed here, because the tooling really is impressive. Letting a model explore a site and draft tests genuinely finds things. The MCP servers that let AI drive a browser are a legitimately useful piece of infrastructure. But as one team building in this space put it bluntly, what you’re getting is browser control, not test intelligence. The AI can operate your app; deciding what’s worth testing, and proving the test actually checks it, is still the hard part. And that part didn’t get automated.
The bill arrives as verification debt
The honest advice everyone lands on is “keep a human in the loop.” That’s valid, but notice what that quietly does to the math. Someone now has to read 200 generated tests to figure out which 40 are load-bearing, which 100 are testing that buttons are buttons, and which 60 look fine but assert nothing. Reviewing a test you didn’t write, to confirm it checks what it claims, can take longer than writing it would have.
That’s verification debt: the gap between “code got generated” and “someone actually confirmed it does what it says.” AI is spectacular at widening the verification debt gap, because it produces plausible-looking output faster than anyone can vet it. The speed you gained at authoring gets quietly handed back at review, and if review gets rushed (and c’mon, it always gets rushed), the debt doesn’t disappear. It just moves to production, where the interest rate is much worse.
What you actually need before you trust the green
None of this is an argument against generating tests. But teams need to demand a bit more than, “it passed” before you let generated coverage anywhere near a release decision. A few questions worth being able to answer for any test, human- or AI-written:
- Does it exercise the behavior, or just the surface? Did it confirm the outcome (i.e. the charge went through, the record saved, the email sent) or only that a page rendered and a button was clickable?
- Why does this test exist? Can you tie it back to a real requirement or business rule? If a test doesn’t map to something the app is supposed to do, it’s decoration.
- Would it fail if the feature broke? The only test worth keeping is one that goes red when the thing it guards goes wrong. If you can’t imagine what would make it fail, it isn’t protecting you.
- Does it go near the edges? Empty inputs, bad inputs, the unhappy path. Coverage that only knows the happy path is a smoke alarm that only works when there’s no fire.
If you can answer those, you’ve got evidence. If you can’t, you’ve got 200 tests and a vibe.
Getting the speed without the slop
The feels-like-magic part of AI test generation is real, and you should absolutely have it. Just don’t let a wall of green stand in for proof. The teams who win with this stuff aren’t the ones who generated the most tests; they’re the ones who can say, with a straight face and the evidence to back it, exactly what their suite protects.
This is exactly the challenge Leapwork Play was built to solve. Play’s fix is essentially, “use AI, but ground the AI in something real and prove what it produced.” Instead of turning a model loose on your UI with a one-line prompt, Play generates tests from your actual requirements, documentation, and existing test assets. The AI isn’t guessing what checkout should do; it’s working from what you’ve said it should do. That single change is what moves generation off the happy path and toward the behavior that actually matters.
Then Play closes the verification gap instead of handing it to you. Every generated step carries an evidence link back to the requirement that justified it, and Play reports, in plain language, why each test was written, and what it covers, so review becomes “confirm this link is right,” not “reverse-engineer what this script was even for.” And because it’s Deterministic by Design, the AI does its thinking during authoring; execution runs as fixed, repeatable logic. No model quietly improvising a different test each run, no live inference silently turning a real check into a hollow one.
