When Your AI Assistant Invents an API

A teammate recently asked me to look at a PR that was failing CI in a way that confused him. The code called ActiveRecord::Base.with_advisory_lock_result — a method with a plausible name, a plausible signature, used in a plausible way, with a helpful comment explaining its behavior. He'd spent twenty minutes checking his gem versions, convinced he had a dependency problem.

He didn't have a dependency problem. He had a fiction problem. There's a gem that provides with_advisory_lock; the _result variant his assistant produced was an invention — a very good invention, which is exactly what made it expensive. It looked like the kind of method that should exist. The assistant thought so too.

This is a genuinely new category of bug for our profession. Humans misremember APIs, sure, but we misremember them vaguely — we go check. Models misremember them confidently, with documentation-quality comments attached. The failure mode isn't sloppiness; it's fluency. And catching it is now part of the reviewer's job description.

The shapes hallucinations take

After a couple of years of reviewing AI-assisted diffs, I've started sorting these into a few recurring shapes.

The plausible method

The flagship. User.find_each_batched, Sidekiq::Queue.pause!, response.parsed_json. These are linguistic interpolations — the model has seen find_each and in_batches and blends them. They feel right because they're built from real morphemes of the API. The tell: you can't quite remember seeing it in the docs, and neither can the docs.

The wrong-version idiom

Subtler and more common. The code is real — for a different major version. Rails 5 patterns in a Rails 7 app (update_attributes, anyone?), old Sidekiq middleware APIs, a Ruby 2.x stdlib call that was extracted to a gem in 3.x. Training data spans a decade of tutorials, and the model doesn't reliably know which decade your Gemfile lives in. These sometimes even run — deprecation shims are generous — and then break on the next upgrade, as a small gift to your future self.

The imaginary option

The method exists; the keyword argument doesn't. validates :email, uniqueness: { scope: :account, case_insensitive: true } — close, but the real option is case_sensitive: false. Many libraries silently ignore unknown options, which upgrades this from "error" to "code that runs and doesn't do what the reviewer thinks it does." I find these the scariest of the bunch: green tests, wrong behavior.

The phantom gem

The assistant confidently requires a package that doesn't exist, or names a real-sounding one. Beyond the wasted time, this has a genuine security angle: people have registered packages matching commonly hallucinated names, so "install whatever the error message says is missing" is no longer a safe reflex. Look the gem up — repo, downloads, maintenance — before it goes anywhere near your Gemfile.

The wrong-framework transplant

Express idioms in a Rails controller, Django-isms in ActiveRecord (Order.objects.filter(...) has genuinely crossed my screen), RSpec matchers in a Minitest file. Usually loud enough to catch immediately, but the quiet versions — a subtly Python-shaped exception hierarchy, middleware assumptions from another ecosystem — can survive a skim.

Why review lets these through

Here's the uncomfortable bit: traditional code review is calibrated for human failure modes. We look for logic errors, edge cases, design problems — and we extend a baseline of trust that the code, as written, at least invokes things that exist, because for a human author, typing a method name usually means having seen it somewhere.

Generated code breaks that assumption, silently. The prose is confident, the naming is idiomatic, the comment explains the invented method's behavior in fluent English. Every surface signal we use to gauge "this author knows what they're doing" is present. The signals have simply stopped being evidence.

So the fix isn't paranoia; it's a small shift in stance: existence checks are now part of review. Not for every line — for the load-bearing calls you don't personally recognize.

Verification that takes minutes, not hours

The good news: this is one of the cheapest bug classes to catch, if you actually look. My toolkit, in ascending order of effort:

Ask the runtime — it can't hallucinate. The console settles method-existence questions in seconds:

User.respond_to?(:find_each_batched)        # => false. Case closed.
User.method(:find_each).source_location     # real, and here's where it lives
ActiveRecord::Batches.instance_methods.grep(/batch/)

Read the installed docs, not your memory. ri, bundle open <gem>, or the gem's docs for the version in your lockfile. The wrong-version idiom only gets caught by checking against what you actually run. Memory — yours or the model's — is exactly the thing under suspicion.

Actually execute the path. Not just the test suite — the tests were often generated in the same conversation, and I've watched a generated test mock the hallucinated method into "passing." Boot the console, call the real thing against a real object once. Thirty seconds of empiricism beats any amount of plausible reading. For imaginary options, this is the only reliable catch: run it and observe the behavior, because the code will happily run either way.

Let the tools be pedantic for you. RuboCop catches some vanished constants; Sorbet or RBS-based checking catches far more of this class than dynamic Ruby deserves; CI that runs against the honest dependency set catches phantom gems. None is sufficient alone, but together they turn "reviewer vigilance" into "red build," which is a much better place for vigilance to live.

Interrogate the diff differently. For human code I ask "is this right?" For generated code I add a prior question: "is this real?" Any method I can't place, any option I don't recognize, any gem I've never heard of gets thirty seconds of existence-checking before I evaluate its logic. It sounds slow. It's faster than my teammate's twenty minutes of debugging a dependency problem he didn't have.

The checklist I actually use

When reviewing (or committing) AI-assisted backend code:

1. Unrecognized method or option? Prove it exists — respond_to?, source_location, or the docs for your locked version. Two minutes, maximum.
2. Check idioms against your actual versions. The code being "real Rails" isn't enough; it has to be real for your Rails.
3. Never install a gem straight from an error message or a suggestion. Look it up first: repo, downloads, last release.
4. Distrust tests generated alongside the code — verify they exercise real behavior, not mocks of the fiction.
5. Run the path once, by hand, in a console. The runtime is the one participant in this whole process that cannot make things up.

I'm genuinely not anti-assistant — these tools have taken real drudgery off my plate, and most of what they produce is fine. But "most" is doing quiet work in that sentence. The old reviewer's question was does this code do the right thing? There's now a question in front of it: does this code do anything at all? Answer that one first. It's usually quick, and every so often it saves you from shipping a very well-documented piece of fiction.