How Migrating SourceGraph’s Search Engine Would Change with AI
SourceGraph needed boolean operators in their search. AND, OR, NOT - the basics that let you search for “this function AND that return type” without regex gymnastics.
The problem: their query parser treated searches as a flat list. Boolean operators need an expression tree. You can’t evolve from one to the other - you have to replace the whole thing. This parser handled millions of queries daily from engineers at Amazon, Uber, and Yelp.
Rijnard van Tonder took on the migration. His constraint was blunt:
“I didn’t want to piss off tens of thousands of engineers.“*
He built confidence systematically. Differential testing ran both parsers in parallel, comparing outputs. Production queries got replayed through both versions. Feature flags enabled gradual rollout. The decision framework was refreshingly honest: “timeline and vibes.” After a month of squashing increasingly esoteric bugs, it was time to ship.
This is “unglamorous engineering” (Rijnard’s words, I love it) at its best: the methodical work of building confidence in code you can never fully prove correct.
This migration happened in 2021. When I asked Rijnard how it would be different with today’s AI tools, his answer surprised me: the work would be harder, not easier. Here’s how to actually pull off a migration like this, and why AI makes the playbook more powerful, at a cost.
Building confidence is about breadth
The key insight from Rijnard’s approach: confidence comes from testing broadly, not deeply. Grammar-aware fuzzing found shallow bugs fast, then plateaued. Unit tests caught logic errors, but nothing else. Integration tests verified plumbing. Real-life confidence came from comparing old and new outputs.
Comparing real-world examples was critical, but unlike Slack’s Vitess migration, the comparisons happened mostly locally during development. Rijnard built a harness to collect queries from production that returned different results and set up a test suite in CI to run them. This infrastructure was temporary, meant to be deleted once the migration finished. Historically, only multi-year migrations justified building throwaway test suites like this.
How AI Changes the Migration Playbook
When I asked Rijnard how this migration would be different with today’s AI tools, three things stood out:
Migration-Specific Test Suites Are Now Cheap
Rijnard spent hours writing a script that would take a query, run it through both parsers, and compare the outputs. Today, you’d describe what you want and get working code in 15 minutes.
This changes what’s worth doing. Before, you’d only build throwaway test infrastructure for terrifying multi-year migrations. Now? Build a custom comparison tool for any parser change. Generate a fuzzer specifically for your edge cases. Write scripts to replay last week’s production traffic through both versions. Delete it all when you’re done.
The friction that used to make engineers skip this kind of testing is gone. You can actually test the way you always knew you should.
Every Implementation Is Now Disposable
Before AI, you’d implement something one way and push through because exploring alternatives was expensive. Now? Rijnard found himself in endless loops: “I’ve implemented something this way, but maybe this other thing works… Why don’t I explore some more?”
The parser could use recursive descent or parser combinators or a state machine. AI will happily generate all three before lunch. The cruel part: you still have to pick one and ship it.
The discipline that used to be imposed by friction - it’s too expensive to try another approach - now has to be self-imposed. Teams that can’t resist exploring one more implementation will never ship.
AI Compresses Work, Not Cognitive Load
Rijnard discovered something counterintuitive: removing the mechanical work makes migrations harder, not easier.
“You can move faster, you can execute faster, but you have to keep up with the speed that you’re executing.”
Writing a test harness by hand gives your brain a break. Having AI generate it means you immediately jump to the next hard decision. Decision after decision, back-to-back, with no palate cleanser.
The result is earlier mental fatigue. You get more done, but you’re exhausted faster. The easy work was busywork, sure, but it was also recovery time between decisions. Take care of yourself!
Your Next Migration
Every major migration ends the same way: an engineer stares at test results and makes a judgment call. Not proof, just accumulated evidence. “Timeline and vibes,” as Rijnard puts it.
Here’s what changes with AI: that throwaway test suite you’d never write? The one that compares old and new outputs? The custom fuzzer for your edge cases? Build them all. It takes minutes now, not days.
Use AI to generate your migration infrastructure:
- Comparison harnesses that run both versions in parallel
- Scripts to replay production traffic through both implementations
- Fuzzers targeting your specific parser/API/whatever
- Delete it all when you’re done
The friction that made you skip this testing before is gone. So stop skipping it.
Time-box your testing (Rijnard gave it a month). Define “correct” in concrete terms before you start. And when bugs stop appearing from any angle, ship it.
The unglamorous truth remains: somebody has to decide when it’s good enough. But now you can actually build the confidence to make that call. AI won’t make the decision for you, but it’ll build all the tools you need to make it well.
You can watch the full conversation with Rijnard below. Get new episodes a few times a month by subscribing via email below, on YouTube, Spotify, or Apple.