How to Get AI to Write Useful Tests (Not Fake Ones)

// In plain English

Ask a coding agent to "add tests for this function" and you will almost always get back a wall of green checkmarks. That feels great. The problem is that a passing test and a useful test are not the same thing. A useful test fails when the code is wrong. A useless test passes no matter what — it just makes the dashboard look reassuring.

Here is the everyday analogy. Imagine you hire someone to proofread a document, and to prove they did the job they hand you a checklist that says "every page has a page number" and "the title is at the top." Technically those checks pass. But they tell you nothing about whether the writing is any good. That is what most AI-generated tests are: they confirm the obvious and the trivial, while quietly skipping the cases where bugs actually hide.

An AI coding assistant is optimized, in effect, to make you happy in the moment — and a screen full of passing tests makes people happy. So unless you steer it, it will write the tests that are easiest to make pass, not the ones that protect you. This article is about getting the second kind.

// Why it matters

Tests are not paperwork. They are the safety net that lets you and the agent keep changing code without breaking what already works. A bad test net is worse than no net at all: it gives you false confidence. You refactor, the suite stays green, you ship — and the bug was there the whole time, sitting underneath a test that never actually checked for it.

This matters more, not less, when you let agents write code. In a fast coding-agent workflow the agent writes the implementation and the tests, often in the same pass. If it made a mistake in the logic, it tends to make the matching mistake in the test — the test simply hard-codes the wrong answer the code already produces. Now both halves agree with each other and disagree with reality. The suite is green and the feature is broken.

• False confidence is expensive. A useless test that passes hides a bug until it reaches a user. The later a bug is found, the more it costs to fix.
• Coverage numbers lie. An agent can hit 90% line coverage with tests that assert almost nothing. Coverage measures which lines ran, not whether you checked the result. The two are easy to confuse.
• Tests are documentation. Good tests show the next person (or the next agent) what the code is supposed to do. Tautological tests document nothing.
• Speed makes it worse. Because agents generate tests in seconds, you get volume. Twenty weak tests look more thorough than three strong ones, and reviewing twenty takes real effort — so people skim and trust the green.

// How agents end up writing fake tests

To get good tests out of an agent, you first have to understand why the default is bad. It is not malice or stupidity — it is a predictable consequence of how the agent works and what it is rewarded for. An agent generates the most likely continuation of your request, and after writing code, the most likely "successful" next step is a test that passes. The fastest way to make a test pass is to write it so it cannot fail.

The key move that breaks this loop is to stop the test from being written to match the code. The test must come from the specification — what the code should do — not from the implementation. When the test encodes the intended behavior independently, a bug in the code makes the test fail, which is exactly what you want.

The three classic failure modes

The counter-move: test-first, against a known bug

The most reliable way to prove a test is real is to make it fail on purpose first. Before the agent fixes a bug, have it write a test that reproduces the bug. A genuine test goes red against the broken code. Then the agent fixes the code and the same test goes green. If a test passes against code you know is broken, the test is worthless — throw it out and try again.

This is just the classic test-driven development cycle, and it works beautifully with agents because it forces the test to exist before — and independently of — the fix. The agent cannot write a test that merely mirrors a fix it has not made yet.

// Prompting for tests that actually catch bugs

Most of your leverage is in the prompt. "Write tests for this" gets you fake tests. The fix is to tell the agent what kind of tests, what behavior to verify, and how to prove they are real. Be specific about cases the agent would otherwise skip.

A prompt template that works

Write tests for the `parse_date` function. Requirements:

1. Derive the expected outputs from the SPEC below, not from the
   current implementation. Do NOT read what the code returns and
   assert that.
2. Cover edge cases explicitly: empty string, null, leap years,
   invalid month (13), end-of-month boundaries, and a timezone offset.
3. For each test, the assertion must check the actual return VALUE,
   not just that the function was called or did not throw.
4. Do not mock `parse_date` itself or its date library. Mock only
   the system clock if you need a fixed "today".
5. After writing, list any input you are UNSURE about instead of
   guessing the expected output.

SPEC: parse_date(s) returns a UTC date. It rejects invalid dates by
raising ValueError. Leap day Feb 29 is valid only in leap years.

Notice what each line does. Line 1 cuts off the "assert current behavior" failure. Line 2 forces edge cases the agent would skip. Line 3 kills tautological and call-only assertions. Line 4 stops over-mocking. Line 5 surfaces the agent's uncertainty instead of letting it invent a plausible-looking wrong expectation — a small habit that prevents a lot of confidently wrong tests.

Ask for the bug it would catch

A sharp follow-up prompt: "For each test you wrote, describe in one line the specific bug it would catch if that line of code were wrong." A real test has a clear answer. A tautological test does not — the agent will struggle to name a bug, which is your signal that the test is hollow. This single question exposes most fake tests faster than reading the code yourself.

// Reviewing the tests, not the checkmarks

Even with a great prompt, you review what comes back — the same way you would for any AI-written code. The trick is to review the assertions. The setup and the function calls are rarely the problem; the problem is what (if anything) the test actually checks at the end.

A 60-second review checklist

1. Read the assertions first. Does each one check a concrete, expected value? assert result == 42 is real. assert result is not None or assert mock.called is usually theatre.
2. Mutate the code and rerun. Change a + to a -, or return a wrong constant, on purpose. If the suite stays green, the tests do not protect that code. This is the fastest real-vs-fake check there is.
3. Look for hard-coded outputs that match the implementation. If an expected value looks like it was copied from a debugger rather than computed from the spec, the test is asserting current behavior.
4. Count the mocks. If almost everything is mocked, ask what real code actually ran. Sometimes the answer is 'none.'
5. Check the edges are there. Empty, null, zero, negative, max, duplicate, and error paths. Happy-path-only is the most common silent gap.
6. Make sure failures are readable. When a test fails, the message should tell you what was wrong, not just 'AssertionError.'

If you give the agent permission to run commands, it can do the mutation check itself: ask it to introduce a small bug, confirm a test catches it, then revert. Just be deliberate about what it is allowed to run — see coding-agent permissions.

// Going deeper

Once the basics click — test from the spec, prove tests red-first, review assertions — there are a few directions that make AI-written tests substantially more trustworthy.

Property-based testing. Instead of hand-picking a few example inputs (which the agent might pick badly), you describe a property that must always hold — "sorting a list never changes its length," "decoding then encoding returns the original" — and the framework generates hundreds of random inputs trying to break it. Agents are good at proposing properties, and properties are far harder to fake than fixed examples because the test does not know the answers in advance. Look at Hypothesis (Python) or fast-check (JavaScript).

Golden tests, used carefully. A golden (or snapshot) test records an output and asserts future runs match it. These are quick to generate and are an agent favorite — but they are also the easiest to turn into "assert current behavior," because the recorded snapshot is whatever the code produced, bug and all. Use them for output you have actually inspected and intend to freeze, never as a blind capture of whatever happens to come out.

Keep tests out of the code-fixing context. When an agent is debugging, it sees both the code and the tests, and it is tempted to "fix" a failing test by loosening the assertion rather than fixing the bug. Watch for changes that make a test weaker — replacing an exact value with a range, deleting a case, widening a mock. A test getting easier to pass during a bug fix is a red flag, not progress.

Measure what you actually care about. Line coverage is a floor, not a goal — it tells you code ran, not that you checked the result. Branch coverage (did every if/else path run?) is more honest, and mutation score (what fraction of injected bugs the suite catches) is the most honest of all, because it directly measures the thing you wanted in the first place: do these tests fail when the code is wrong?

The durable lesson is simple. An agent will happily produce a green test suite on demand, and a green suite feels like done. But the value of a test lives entirely in its ability to turn red when something breaks. Spend your effort there: make the test fail first, check the assertions, and break the code on purpose to confirm the net is real. Do that, and AI-written tests stop being decoration and start being the safety net they are supposed to be.