DEV Community: Gil Zilberfeld

Knowing the Words Isn't Knowing the Language

Gil Zilberfeld — Wed, 07 Oct 2026 07:42:51 +0000

You've got a 200-day Duolingo streak. The owl is proud of you.

Then you land in Rome, sit down in a trattoria, and order. Every word is owl-approved. The grammar too. You think.

The waiter nods, smiles, and brings you something you don't really want. Is beef cheeks a thing?

Knowing the words isn't knowing the language. Even with a 200-day streak.

Code is made of words. Words that computers understand. If we could talk to computers with natural language, everything would be great. The problem is not with the words we speak, it's with what the computer understands.

And they understand ones and zeros. That's not going to change.

How do we get from natural language to zeros and ones? We use "code". Our code is an abstraction above ones and zeros.

As any tourist can tell you, knowing a few keywords is not enough to convey the meaning of what you want, and reach the outcome you want. The better you speak the language and know the nuances, the better results you'll get.

Let's look at the process of "getting results". I'll keep it simple, because in the real world this is a lot more complex.

Requirements -> Design -> Code -> Build -> Verification

Each stage includes a translation.

Requirements – The translation of what the customer wants to what to build
Design – Translation of what to build to how to build it
Code – Translation of how to build in our words to computer commands
Build – Translation of computer commands to zeros and ones
Verification – Translation of what we see back into what we meant to build

With each translation there are translation errors. We know that, so we put processes in place to plug the leaks. Requirements review, design reviews, code reviews, automation tests, canary releases.

We know there are going to be problems, so we try to minimize the risks.

Final point – except for the translation from computer commands to ones and zeros – we control (or think we do) everything.

So, is AI making code worse?

Not necessarily. I've seen good AI code that is better than some developers' code.

But that's not the point. The code is a translation, this time by a genie, whose way of thinking we can't question or understand. Because it doesn't really think.

We get a dish. But not exactly as we wanted it.

And when we start delegating requirement specification, design and verification to AI, we're losing more and more controlling points.

The outcome is mostly ok, because coding agents get better. But we all know the big problems fall through the cracks.

We all know about security problems and performance issues introduced by AI code. But even the functional stuff suffers because of unintended side effects, weird behavior, and other decisions made by the agent.

If the code worked perfectly, we wouldn't ask the question. But we know we'll need to get in there and fix stuff, and the code will resist. And it will be painful, especially if it's the first time we see it.

And improve code you've never read? That's risky business.

So what can we do?

Review, review and review. Not just code. Docs, plans, anything the genie produces. You can't fix what you haven't read.
Work in small chunks. Our review capabilities are a lot more effective in small chunks. That's also about finding errors in code and architecture. For us and the genie.
Refactor. Once you've read it, if you don't like the code, change it. Tell the genie how to do it better the next time.

And yes, code gets worse. Just because we don't improve it. No matter who wrote it.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

The Test Was Wrong. Rewriting.

Gil Zilberfeld — Wed, 30 Sep 2026 13:21:18 +0000

Did you ever see this line when you let your code agent run wild?

“The test was wrong. Rewriting.”

Can the genie really make mistakes?

Ok, enough jokes.

If you’re surprised, this may be the first time you’re generating tests. It happens a lot – seven times in a single build in one of my checks.

Let’s walk through a couple of scenarios, and see how we got to this place.

For our purpose, let’s assume that we generated both code and tests. But the analysis is true for when the code agent generates just the test.

How did we get here?

The genie generates test code. This code is just lines of text in a file – unproven until it runs. It can be proven wrong syntactically, and by some heuristics, logically. Meaning, knowing it’s wrong without running it.

So if the genie runs this check only (we don’t really know unless it decides to tell us), it’s a guess. Not 50-50, but still not a real proof.

Our genie is nothing but a truth chaser. Rightly so, it runs the test. If the generated test fails, we have a fork in the road.

The code is right, and the test is wrong.
The code is wrong, and the test is right.

Nah, it’s not that simple. Both can be wrong. It can also be:

The genie understands the test doesn’t even run the code. Or,
The genie realizes the test isn’t designed correctly to check what it needs to.

Happened to me, where a test that should have recreated a race condition turned out to never run the race at all.

I want you to understand that at least in one of the stations, the genie made a mistake.

Creating the code
Creating the test
Validating the test vs the code
Running the test
Evaluating the test for its purpose

#4 is the easiest to spot. Why?

Because it’s not reasoning. It’s a deterministic run – pass or fail. So a failure means something is definitely wrong.

In all other stages, whatever the genie does with the code is

a) Hidden from us

b) Non-deterministic, meaning we got the result today, but maybe if we ran this yesterday we’d get a different answer, and

c) It actually caught something. Or thinks it did.

Everything except the actual run is opaque and not repeatable. Not what we’d call proof.

A crime may have happened here. Or not.

This is not an anti-genie rant.

Let me be honest. I don’t – I can’t – read everything it logs. I run the agent, and look at the end results. Tests passing? Cool. I usually don’t look back at the logs.

Heck, the race condition issue? I asked Claude to go through the build logs to find where tests were found wrong.

I use coding agents all the time. And looking at me, you’d say – I trust them. Because we were taught that trust looks exactly like this.

But, this is not trust. It’s a bet. Lots of them. The app does work mostly, and when I find something I ask for a fix. And if the agent finds something, it fixes it.

But remember the options? The genie can make mistakes. Also in fixes. And in new code. And in replacement tests.

Who says the fix is the correct one?

My app, even in production, does not carry the risks of fully scaled apps. Imagine hundreds of developers, each with their own agent making those bets on your finance apps. Or law practices. Or online election management.

Are you scared? I know I am.

The way out is chunking the tasks to be smaller and manageable. Smaller pieces of code produce smaller logs. Ones we can read better.

Reviewability – if it’s not a word, it should be – is now a delivery capability.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

We Finally Got the Tests. Now We Don’t Trust Them.

Gil Zilberfeld — Wed, 30 Sep 2026 13:20:23 +0000

Testers have been badgering developers for years to write unit tests. Now even those devs who don’t like tests can generate tests.

So now testers say, well, we don’t trust these tests. And they have good reason, too. Unit tests may pass, while the code is broken.

First, let’s explain the badgering. We don’t have a lot of time. Didn’t before AI, and definitely not now.

So when we get our hands on the app, or the new features you created, dear developer, we want it as stable as possible. We don’t want it to break the moment we touch it.

Now, one way is to check every build from scratch. But like I said, we don’t have time. If we have proof that you checked the code in some way, that means we don’t have to waste time on it.

So you’ve got tests? That’s proof. Thank you!

Except…

Well, where did the tests come from? Ah. Generated. I see. Hmm.

And I assume there are no bugs in the code, right? Because, if there are, and you’ve got some tests covering that bug, then you and I are pretty screwed.

What do I mean? Here’s an example.

Here’s a function that checks the expiration of a credit card.

def is_expired(exp_month, exp_year, today):
    return date(exp_year, exp_month, 1) < today

Looks ok. But not really. Credit cards have MM/YY expiration, and that means that they do not expire until the END of the month.

This code has a bug in it. It causes a card to expire on the 2nd of the month.

No worries, a test will catch that bug, right? Here’s what my genie created:

def test_card_expired_last_year():
    assert is_expired(9, 2025, date(2026, 9, 16)) is True

def test_card_expiring_this_month():
    assert is_expired(9, 2026, date(2026, 9, 16)) is True

The first one’s great. It checks that a card with 09/25 expiration is indeed expired.

The second one is weird. We know the card should not expire. It’s not the end of the month yet, so we expect the result to be False, not True.

Why did it generate the wrong test?

Well, if you generate tests for the code, you get tests that pass for the code. Not for the requirement. This is called a Tautological test – it confirms that the code works as written. But not as it should really work.

What the genie didn’t have is context. So it guesses when it generates the code, filling holes in what it has been told, and what “good code looks like”.

But without context (and let’s face it, sometimes with it) the code may be wrong.

The bigger problem is that now the tests make it look like it works as intended.

And what happens next? The code breaks when we touch it.

That’s why testers have trust issues.

One more thing – if a human read the code, chances are the bug could have been caught in the review.

So what can we do?

Ask which tests are generated and which we can trust.
Ask what code is reviewed by a human.
Know where the risky logic sits and test it extensively.
Spot the passing tests (that shouldn’t pass) and fix them.
Teach the devs to do that.

The irony is that while we finally got more tests, our confidence in them is lower.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

In a Few Weeks, I Won't Know How My Own App Works

Gil Zilberfeld — Wed, 09 Sep 2026 12:53:49 +0000

I’ve been doing a lot of code reviews recently, and I’ve noticed something about my experience during the reviews. And what it means for code in general, but specifically for AI generated code.

But before I do that, I want to take you back to the ’00s. A simpler time. Back then I started doing something I only read about in books – pair programming. Ok, that’s a lie. Everyone who grabbed someone and brought them to look at your code because it’s doing something weird – you’ve done pair programming.

But this was pair programming for like 80% of my work. And of course, not just me. This links directly to my topic – our understanding of the code. And in the age before AI, where teams really knew their code, that would be the biggest grade I’d give for code understanding.

Meaning, if you and I work both at the same time on the same task, and finish it. I’d say we’re both at the highest level of understanding of that feature – what it needs to do, how it works, what it depends on, what depends on it, and how it’s written and tested.

From here that level of understanding is going to drop. If you’re part of a team, but didn’t work on that feature, you probably know it exists. If we have dependencies, you probably know about them. But the rest is either non-existent or close to it.

If you’re on another team, that level of understanding drops, and if you’re on another project – you may not even be aware.

That makes sense, but we’re not talking about that today, I want to focus on my capability to do a proper review. And I go back to understanding for that. In order for me to give you proper feedback, catch your mistakes and offer alternatives, I need proper understanding. The more understanding I have, I can review better and make better changes.

So far, so good. It makes sense.

Two more things make sense. First the use of tools for review. Every tool, from syntax analyzer to the smartest security checker, work according to patterns, not understanding. All the tools check for generic stuff. Newer tools that use LLMs, can find more elusive patterns. But in the end, it’s like finding a missing semi-colon.

The second thing is async code review. Or PR review, as people call it. This is done not with the person who wrote the code, but at my spare(?) time. I pull the PR, look at the code and based on my understanding, I need to approve it. Note that at this point, I can’t ask questions, so guess what I do? Look for patterns. If I’m the team leader and I know the feature, I can review better. If I’m on another team – I will not. Not take risks. Not initiate another coding and code review. Who has time for that?

And that brings me to reviewing code my genie wrote.

Because if you think about it, I’m the team leader, and the code agent is on my team. I’m supposed to know more about the feature, but I’m reviewing code I didn’t write.

How well can I do it? There’s another part that contributes to this, which is the size of the diff. Humans, before AI know their limitations (most of them), and so they don’t write a lot of code, because they know what it’s like to review a lot of code. But even if they do, the reviewer, unless she gives up, can still ask questions.

Well, my genie does a lot of work and creates a lot for me to review. Now, I know I can’t review a lot of code in one go, so I make my genie work in chunks. Chunks I can read and understand. For example I break features into workflows, and ask the genie to do a workflow or two, including all tests and whatever’s needed.

Which is still a lot. And I tried to explore what happens that makes me give up. Yes I do give up sometimes.

It goes back to understanding. I’m reviewing pieces of code – tests, bits of FE, bits of the API code, bits of logic and bits of database code. I can review these at the lowest level – does it do what I need? Can I read it?

But when it comes to bigger things – like dependencies and side effects and performance – I give up. First there’s a lot of code to absorb, and then I need to piece everything that I know (or think I know) of how the rest of code works and designed. I’m building a code model in my mind. This helps me identify design issues, find the side effect traps. Maybe the performance issue hiding in there.

But, if it’s hard to build that puzzle, I’m not going beyond “does it look ok” and “do the tests run”.

So, I’m taking risks. Risking that there’s code in there that I don’t know what it does, how it works and what effect it might have on the system today. And tomorrow, I won’t even consider it for any co-dependence with my new feature, because I don’t understand how it works.

This accumulates over time. Within a few days or weeks – I won’t know how most of my app works.

So what do I actually do about it?

Two things, and neither of them is a fix.

The first one you’ve already seen. I keep the genie working in workflows, not in features. It makes the diffs smaller, and keeps them inside a context I can understand. The second one is a decision. Before I review a chunk, I decide what is important for me to go deep into, and what to go over lightly. Or not at all.

It doesn’t come close to the old level of understanding. But my overall feeling of understanding has improved.

Now, take that, and think about teams and groups. How much code and tests that nobody knows and understands can be accumulated in one year. Let alone three.

Do you understand this risk? And how are you handling it?

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

Testability Is a Feature. Does Your Code Agent Know About It?

Gil Zilberfeld — Wed, 09 Sep 2026 12:48:21 +0000

I always say that testability is a feature.

It needs a customer – usually testers. We need to define what it means. And we need to build it in.

The question is what happens to testability when someone else builds the code.

Yup. Genie talk again.

Let’s look at just 3 aspects of generated code, or any code really, that impact testability. But when our genie doesn’t get directions, it will skip those and cause us a big headache.

1. Complexity

Of course, complexity affects testing. The more complex the code, the more work it takes to verify. More cases, more time – time that we don’t really have.

My classic example is recursion. Recursion seems simple (in code lines), but it can hide all kinds of bugs in there. Edge cases galore.

But that’s usually a function. That can be confined and handled. Today we’re generating systems.

And they are as complex as the genie wants. Or the systems it was trained on. And those are not a model for simplicity.

Complexity is the nemesis of testability. And unless we ask for simplicity, and make sure we got it, we pay for that in more expensive testing.

2. Observability

Observability is a key part of testability. Without it, we may be able to operate the system, but may not see the impact of those operations.

For example, if you had one POST API in the world that saves data in the database, and you want to check if it works, you’ll need to choose between looking inside the database (which may not be possible), or rely on the status code (which doesn’t tell you anything about the actual impact). Adding a GET to read helps, and presto – you’ve got observability.

Of course, you need to ask for that GET API.

In CRUD systems, you’d get that API, I’m not worried. Even in generated code. The problem starts with the not-so-intuitive stuff. What gets logged and where. Knowing how long it takes to see the result, and maybe how much time to wait between operations and states.

The more the system exposes state and data, the easier it is to understand, and the cheaper it is to debug and reproduce issues in it.

Now – what are the chances you get the things you need for your non-generic needs, from a generic code generator?

Exactly.

3. Security

I don’t need to explain how much generated code suffers from security issues. First – not new – it comes from the code examples out there, which are not that secure.

But here’s one thing we don’t talk about. We usually plug security holes by tools. They scan the code and find patterns that are possible entry points.

But now, we have a problem. First – the tools are ok, but they now also use models. Which are not perfect, they created the code in the first place. Plus, we use them to fix the code. Again, the same tools that created the code.

This is where I get trust issues.

But it’s not enough to call APIs and see the result. We need to plan and design cases to identify security issues. The impact of these issues are inside different systems – our app, the database, internal network, the billing system.

Even if our app is testable (and we know how often that’s true), what about the extended environment? How easy is it to design security cases, run them, and see the results?

The risks of code we don’t know don’t only increase the impact of what we know we need to check. They increase the hidden security issues too.

Ok, the devs use those code agents, not you. What can you do?

Talk to the developers. If you don’t, that’s a good line to start with: “Hey, I heard you use code agents”. Works great at parties.
Ask to see the code. Even non-coders understand simplicity. And they can see and understand reports from scanning tools.
Explain how you’re going to test the app, and wait for the “you can’t do that” moment. This is where you make your request.
Define what testability means for you and communicate it to them. They can ask (they would call it “tell”, but don’t fall for that trick) the genie to generate the code with those guidelines.

I’ve said it before, AI amplifies problems and solutions we had before. We need to adapt to the quantities and new risks.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

AI vs CI: Nobody Told the Pipeline

Gil Zilberfeld — Thu, 20 Aug 2026 11:17:28 +0000

CI is so commonplace these days, we don’t often think about why it’s there. And that reasoning really matters these days, because of how we use it.

Especially when you’re writing AI agents and expect the CI system to run a regression test suite, and sound the alarm when something goes wrong.

Here’s a funny story you may not know. CI - Continuous Integration – was not originally a tool category. It’s a process - it’s how you continuously integrate pieces of code together.

Anyway, why do you need this process anyway? Because you’re scared. You’re terrified that integration broke something.

So what do you do? You look for approval. An automation system that runs all the tests all the time is our approval of choice.

That’s how CI became a tool category. Because in its heart, it’s a simple automation pipeline.

So, that automation’s holy grail is speed, right?

Wrong.

The mission was to kill “Works on my machine”

Remember when we wrote software in a cave, and we had a sticker on our computers – “Works on my machine. Don’t run it on another cave”?

Ah, the good old days.

Because the CI automation #1 OG mission was to kill “Works on my machine”. How? By creating a repeatable process. Independent of all the weird installs on my machine. Multiple versions of libraries. And different configurations. And admin hacks.

The CI builds, packages, runs and tests the software every time the same way. That way, if a test turns red, you know something bad happened, because it was green until now. And why your stomach turns, when it flakes between red and green – that’s the feeling of lost repeatability.

Yes, repeatability brings bliss. Well, it did, before AI.

Then AI showed up

If your app does not touch AI (although building with code agents, can sometimes count as “touch”), you’ll continue to feel that bliss.

But if you’re developing AI-based features, or agents, or vibe-coding, you’re in for a new experience every time you push your code (or prompts) into CI.

Things are not repeatable anymore. Models change, sometimes without warning. And always under your feet. And when a model changes, does it trigger a run?

You wish! Your alert system is malfunctioning.

And the worst part is not a regression. Although if it’s a bug in a prompt, you may not be able to fix it.

No, the worst part is that Green today doesn’t tell you what kind of Green it is. Things are no longer Working or not, they are Work-ish. But even that’s not the same work-ish every day.

So we’ve got fewer triggers, results we can’t read, and on top of it – every run means something else.

What’s the solution?

Repeatability gives us confidence. CI is the process, and tools, that give us the approval we seek, and the confidence we have comes from that repeatability.

We can’t rely on the trigger anymore, but we can initiate our own runs. And since one run is not enough, we need to run more and look at the trends. Catch drift before it ships.

What happens to quality when AI meets CI? We need to get back to the reasoning, and change how we manage quality.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

It's Almost Like You Need an Actual Programmer

Gil Zilberfeld — Thu, 13 Aug 2026 07:07:33 +0000

Vibe coding ain’t what it used to be.

Back in the old days (2 years ago), we conjured up apps with our own programming language.

Turns out, it does work, but more for prototypes. If you want an actual working production app, you need that thing from before. How do the old timers call it? Ah yes. Code.

So our builder says to the genie: “Code me this”.
And the genie does build this. Mostly.

When we’ve got a prototype and it’s working, two things happen. The first thing is management thinks they have an actual product. In our case, it’s the builder that thinks “code me this” is simply a translation layer.

But then, another thing happens – we now have to take care of our puppy. I mean the code.

As long as we don’t need to touch it, it doesn’t matter what it looks like. But then we do.

So what does our hero think? What worked before, will work again. He tells his favorite code agent – “build me my greatest feature yet”.

But then weird things start happening. Bugs that weren’t there. Multiple tries of code changes – costly ones, mind you. Performance slows down.

It’s almost like you need an actual programmer to do the work.
And you do. You know what helps developers? And coding agents?

Clean code. Readable, maintainable code. The classics are still with us.

Now, coding agents are getting better at producing code. However, they are only as good as the code they are trained on. Now think – what do you think the average level of code out there is?

Exactly. That’s what you’re getting.

So if you are on a transition from prompts to code, it may look like continuing vibing is the shortest way to more features. Yet, the first thing you need is a good platform to grow your code.

Now it’s time to take control.

Read the code.

You’ll do this first anyway. The code is the only truth you have. The prompts you used before are scattered across forty chat sessions and there were never any docs.

Just know what you’re up against. Agents drop a lot of code, and after ten files your eyes glaze over. You won’t read all of it, but try to understand the important bits.

Write down what it should do.

Ask the agent to derive documentation from the codebase. It’s fast and it beats no docs at all.

But know what you got. Those docs describe what the code does, not what it should do. If there’s a bug in there, it appears as a requirement. Review and sieve out the wrong stuff.

If you can find them, collect your prompts. They were the intent for building the app.

Review both, and drop everything that doesn’t add up. Now you have a documented system.

Now write tests. Against the requirements.

It’s tempting to generate tests for what’s already there. And for legacy apps, I recommend capturing existing behavior of undocumented knowledge built up for ten years.

But here? You’re better set. The intent is still around. Capture what the code does today and you’ve just made every mistake a requirement.

Then you can change it.

Including asking the genie to change it for you – which is the part everyone wants to skip to.

But now, with documentation and tests in place, now it’s a lot safer to make changes.

Next you need to understand the principles of Clean Code. If you don’t, you may ask the genie to change the code for you, but you’ll get more of the same. We don’t want more of that mess.

Exactly like programming. Who knows, maybe you’ll become one eventually.

So what do you say – Are you ready for clean up?

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

The New Technical Debt

Gil Zilberfeld — Thu, 13 Aug 2026 07:06:32 +0000

Is AI code debt the new technical debt? As I keep learning, generating code and examining it – it’s worse.

In the old days, we worked hard to create technical debt. Just kidding, it was easy.

Technical debt has many definitions, and origin stories. But one thing is common: We leave the code as it is, knowing it could be better. The debt is the gap of effort of making it better.

And why the gap? Because we know we’ll see that code again. And when we do, it’ll be hard to change. Better code would have made it easier.

Now we’re in the age of genies. They can write any code. They can change any code. Do we need to worry about AI code debt?

Sure we do.

AI code debt is exactly the same as technical debt – we’re leaving the code as it is, knowing it could be better. The gap is still the effort of making it better.

But this time the gap is a lot bigger. There’s a lot more code, it’s probably a lot more complex, and some of it – let’s be frank – is code we didn’t review. So the gap is a lot bigger than we guess.

But that’s a bot problem, right? The code agent will deal with all the needed changes. We don’t need to even look at the code.

Nah, you know you will. You know what code is generated, and it’s not how you would have written it. And coding agents have the same problem of making sense of complex code bases as us.

And they’ll make mistakes.

Can the code agent clean the code up by itself?

Cleaning is not just “make it readable”. It’s preparing it for more changes, reducing dependencies and isolating interfaces – all in the context of future plans. And the agent doesn’t have this context.

In fact, it will assume another context, and we get stuck with the code and the assumptions.

How do you pay down AI code debt?

Same as always: find the code you know you’ll touch again, and make that code easier to change. The problem starts when you don’t know where that code is.

Because, you didn’t write it. And you didn’t review all of it. If any.

Putting a price on technical debt before was an exercise in imagination. Now it’s pure fantasy.

How do you avoid it next time?

First know what code was generated. And for that you need to enforce smaller code generation.

If you don’t enforce it, you’ll have a lot more to review. And if that happens you won’t review it all. It’s a human thing.

Then, you can wish the genie to refactor it to take the shape you want. Genies are good at transformations. And if the genie breaks something, your tests will tell you.

You do have tests, right?

The old technical debt was based on maintenance work. Now we have bigger maintenance queued up, along with risks of unverified code. Not cool.

So, first, we need to be aware of the AI code debt – we’re creating code that will cost a lot more to maintain, than “regular” code.

Before, we thought “it’s ok, we’ll take care of that later”, and then, when the bill came it was a lot more than we thought.

Now? Expect a much bigger one.

Questions people ask about AI code debt

What is AI code debt?

AI code debt is technical debt created by generated code: you leave the code as it is, knowing it could be better, and the debt is the effort of making it better. The difference is scale. There’s more of it, it’s more complex, and some of it was never reviewed.

Is AI code debt different from technical debt?

Not in kind, only in size and visibility. The old debt was something you put there and remembered. This debt arrived while you were reading something else, so you don’t know where it is.

Can an AI agent clean up its own code debt?

Not on its own. Cleaning up means preparing code for changes you plan to make, and the agent doesn’t have that context. It will assume a different one, and you’re left with the code and the assumptions.

How do you check that an AI refactor didn’t break anything?

With tests you wrote before the refactor. Directing the genie to reshape code is fine as long as something independent tells you when it breaks.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

You Can't Fix A Bug With A Prompt

Gil Zilberfeld — Thu, 06 Aug 2026 10:46:27 +0000

Let's talk about fixing bugs. My second favorite activity after creating them.

How do I know the bug is fixed? Well, I've been taught by wiser and more experienced people than me (way back when), that if you want to make sure the bug never shows its face again - write an automated test for it.

Ok, I admit that there were times I didn't write a test. Some fixes are so trivial, that sometimes a test is a luxury.

But even then, I was working on a hidden assumption. Just like in Fallout, code never changes. Until it does. But between those times it never changes.

I fix the bug. From this point until it actually changes by a human, or a bot or someone in-between - that code will compile, or transpile, or whatever we call "runs" the same way. A statement will execute, a condition evaluated, an exception caught - every time in the same way.

And now we come to modern programming languages: Prompts. Agents, prompts, workflows - they don't run the same way every time. What they do is run their interpretation of the request. If they run tools - they run the deterministic parts. If they run sub-agents, they run interpretations based on other interpretations.

Interpretations work mostly the same way, until they don't. When I was teaching my agent to develop in TDD, I had a couple of requests. One of them was to not create code without a test.

Which I thought was a very normal request from an agent. In fact, when I started out by telling it - we're working in TDD - I assumed it knows what TDD is. Ha.

Always remember that what LLMs know is exactly what most of the population knows. And usually the "don't create any code without a test" falls through the cracks.

So I made it official: One of the agent rules was exactly that. This was a legitimate bug fix. At least I thought so.

But it really was a suggestion. Which the agent considered, and depending on its mood, sometimes did, and sometimes didn't.

I won't go into the full solution (still in progress, if you want me to elaborate, comment), but part of it was to run a targeted coverage tool - a deterministic one, check it and stop the process if it found extra code. The TDD sequence looks like this:

Per step:
1. Write test(s).
2. `node scripts/tdd.mjs red <test-file>` — all newly added tests must fail. Pre-existing passing tests in the same file are allowed to remain passing.
3. Implement minimum to pass.
4. `node scripts/tdd.mjs targeted <test-file>` — targeted coverage + pass check.
5. `node scripts/tdd.mjs lint` — ESLint.
6. `node scripts/tdd.mjs full` — full suite + coverage.
7. Script says STOP. Human reviews.
8. Human runs `node scripts/tdd.mjs commit "message"`.
9. Human confirms next step. AI runs `/clear`.

But each step here is the suggestion. The real enforcement is done in the tdd.mjs code. Real code.

But this is just an example. The real problem is that more and more "code" is not programmed. It's interpreted. That means that bugs are a lot more flaky to catch, but also are not permanently fixable.

And don't get me started on companies switching model capabilities every other Tuesday. In the past, updating versions was a whole ceremony because we were worried something would break.

Now LLM providers do it for us without us knowing.
We can live with that. We should make sure we know.

And remember - bug fixes in prompts are not real bug fixes. They are more like wishes. Which may or may not come true.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.