Reference-First Debugging: Get the Right Answer Before Chasing the Error

The Bug That Wouldn't Die

It's late, I'm staring at a log line that says my swap reverted with an error code I've already seen five times tonight. Each time I patch the thing I think is wrong, the bot runs again, and a new failure shows up two minutes later. Different error, same underlying mood: I have no idea what is actually happening on chain, and I'm pretending I do.

This is the moment that taught me the single most important habit I've picked up while building a Solana MEV bot from scratch. Before you debug anything, get the reference. Find what the correct answer should look like, in detail, before you go anywhere near the broken thing. If you can't describe "right" in concrete terms, you have no business calling something "wrong."

This post is a rant, a confession, and a methodology rolled into one. The methodology has a name in the academic literature, but the lesson reached me the hard way: through hours of patching the wrong symptom in the wrong layer of a system I didn't understand yet.

Why Debugging Eats So Much of Our Lives

Debugging is expensive in a way that most non-developers don't understand. A 2017 ACM Queue article titled The Debugging Mindset cites estimates that software developers spend 35 to 50 percent of their time validating and debugging software, and that the cost of debugging, testing, and verification accounts for 50 to 75 percent of total software project budgets — amounting to more than $100 billion annually in the United States. Whatever the exact number, the rough shape is the same: most of building software is not writing it. It's figuring out why what you wrote doesn't behave the way you thought it would.

In an MEV bot the cost compounds. Every minute the bot misbehaves is a minute it's either burning fees on losing trades or missing real opportunities while you stare at logs. The market doesn't pause while you debug. So the cost of a bad debugging habit isn't just engineering hours — it's measured in lost edge, in days where the bot loses money it didn't have to lose.

And it gets worse in the era of generative coding assistants. The 2025 Stack Overflow Developer Survey, summarized in Syncause's Industry Survey, reported that 66 percent of developers found AI-generated code was "almost correct, but not quite," and 45.2 percent said debugging AI-generated code was more time-consuming than debugging human-written code. The Syncause writeup also references a 2025 METR randomized controlled trial in which experienced developers took roughly 19 percent longer with AI assistance, despite expecting a 24 percent speedup.

Faster typing, slower thinking. That's the trap I keep walking into. The bot makes a swap that fails on chain. An assistant offers three plausible-looking fixes within seconds. I pick one, push it, and now I have a new failure to debug — except now I also have someone else's code in the middle of my mental model, and I'm even further from understanding what the system was actually trying to do.

What Experts Actually Do Differently

In 1986, a study published in the ACM SIGCHI conference proceedings compared expert and novice programmers debugging the same code. The numbers are striking. Experts succeeded 89 percent of the time; novices, 72 percent. Experts averaged 7.0 minutes to find a bug; novices averaged 18.2 minutes — more than twice as long.

The interesting part isn't the time gap. It's how the gap formed. Novices, the researchers found, read code linearly, top to bottom, like reading an English newspaper, regardless of the actual execution order. Then they immediately jumped in and tried to find the bugs. Experts, by contrast, generated high-quality hypotheses based on less study of the code than novices, used breadth-first approaches, took a system view of the problem, and grouped related code into conceptual blocks before drilling into any specific line. And — this is the line that stuck with me — "experts exhibit metacognition, recognizing when it's necessary to engage in specific strategies for code comprehension based on their self-awareness."

Experts know when to stop reading and start tracing. Novices just keep poking. The novices in that study also "frequently added bugs to the program during the course of trying to find the original one." Sound familiar? It's the patches-on-patches death spiral. You touch something, break two other things, touch those, break two more, and at 2 a.m. you can't remember what the original bug even was.

What the experts had — and what the novices didn't — was a mental model of what the program was supposed to do. They had the reference.

The Reference Is the Answer Key

Think of debugging like grading a test. If a student writes "the capital of California is Los Angeles" on an exam, you can only call it wrong because you already know it's Sacramento. Without the answer key, you don't know if a single response is right or wrong — you just have a bunch of student responses. You can't grade anything.

Debugging works the same way. The error message in front of you is the student's response. The reference — what the system was supposed to do — is the answer key. If you don't have the answer key, you're not debugging; you're guessing. You're a substitute teacher staring at a stack of papers in a subject you've never taught, marking things wrong based on vibes.

In Andreas Zeller's foundational book Why Programs Fail (Morgan Kaufmann, 2005; O'Reilly listing), this is framed as the chain from defect to infection to failure. The programmer creates a defect in the code; when the code runs, the defect causes an infection in the program state; that infection eventually becomes visible as a failure. What you see in your logs is the failure — the very end of the chain. Debugging means walking backward, from failure to infection to defect. But you can't walk backward unless you know what the correct chain — the reference chain — was supposed to look like at every step.

This is why a 1997 paper by Marc Eisenstadt in Communications of the ACM, Tales of Debugging from the Front Lines, based on debugging anecdotes from 78 commercial software implementors, found that "programmers often identify the intended state of a program before they find the bug." That sequence matters. Intended state first. Bug second. Always.

Eisenstadt also found that debugging was at least twice as efficient the second time programmers debugged a program, even when it was a different bug. The reason is obvious in retrospect: by the second pass, they had a mental model. They had built their own reference. Everything after that was comparison, not exploration.

Rule One in Every Serious Debugging Book

If you read the canonical books on debugging, the reference-first principle is sitting right there in rule one of almost every list.

David J. Agans wrote a small classic called Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems (debuggingrules.com, AMACOM, 2002). The book's framework applies to software, hardware, plumbing, cars, furnaces — anything that can break. Agans's Rule #1, before anything else, is Understand the System. "Read the manual," he writes, "read everything in depth, know the fundamentals, know the road map, understand your tools, and look up the details." Rule #3 is Quit Thinking and Look: "get data first rather than doing complicated repairs based on guessing. If you guess at how something is failing, you often fix something that isn't the bug, and not only does the fix not work, but it takes time and money and may even break something else."

That second paragraph might as well be a portrait of every bad night I've had with the bot.

Nicole Tietz-Sokolskaya's 2023 post A Systematic Approach to Debugging lists six steps. Step three is "Understand the system(s)." She writes, plainly: "This one sounds obvious but it's a step people skip a lot." Of every step in her list, she calls this one out as the one developers cut. Not because it isn't important — because it isn't fun. Reading docs feels like procrastination. Patching feels like progress. It isn't.

The MIT 6.102 Software Construction course frames it as a scientific method: study the data, hypothesize the cause, design an experiment, repeat. Their key principle: "Don't Fix Prematurely: Probe first to understand root cause; avoid ad hoc patches that mask symptoms." Cornell's CS312 lecture puts it even more bluntly: "Testing reveals the effects (or symptoms) of errors, not the cause of errors." You can't distinguish symptom from cause unless you have a reference for what the cause-free state is supposed to look like.

And then there's the John Carmack quote, surfaced in Jason C. McDonald's 2019 post The Rules of Debugging: "Most bugs are a result of the execution state not being exactly what you think it is." The gap is between what you believe is happening and what's actually happening. The reference closes that gap on the "what should be happening" side. Logging and tracing close it on the "what is happening" side. Without both, you're guessing.

Why This Bites Twice as Hard on Solana

A traditional web bug at least lives inside code I wrote. An on-chain bug lives in a system I didn't write, that I can't single-step through, that updates while I'm reading the docs, and that hands me a five-byte error code as my only feedback.

A failed transaction on Solana doesn't come with a friendly stack trace. It comes with a custom error number, a program ID, and a compute units number. The protocol I'm interacting with may have an IDL, or may not. The exact behavior of an instruction in some specific edge case is often only verifiable by reading the on-chain program's source — and even then, the version of the deployed binary might not match the public repository because of unannounced upgrades.

My first instinct in this environment used to be the worst one possible: see error, search error, paste fix. That works in mainstream web development because the spec is well documented, the error messages are descriptive, and Q&A sites have a decade of accumulated answers. None of that holds for the part of an MEV stack that actually matters. If I had a dollar for every time the top search result for an obscure on-chain error was "I had this too, I solved it by [unrelated change]" — well, I'd at least cover one of my failed bundles.

The reference-first habit forces a different opening move. Before I touch a line of code, I find the part of the on-chain program's source that defines the instruction I'm calling. I read the validation logic. I find every place that exact error code can be emitted. I write down, in plain English, what each of those branches actually checks. Only after I have that list — the reference — do I look at the failing transaction and try to figure out which branch I'm tripping.

The difference in outcome is night and day. With the reference in hand, the failing transaction tells me which check it failed. Without it, the failing transaction tells me nothing useful at all.

The Rubber Duck Knows

Rubber duck debugging — the practice of explaining your code line-by-line to a literal rubber duck on your desk — gets joked about because the image is silly. The mechanism is dead serious. As Wikipedia's entry on rubber duck debugging (drawing on the 1999 book The Pragmatic Programmer by Andrew Hunt and David Thomas) describes it: "Because the subject has no context, the developer must explain both what the code is supposed to do and what the code on the screen does. In meticulously explaining both the expected function and the actual code, incongruities between the two become clear."

Notice what that mechanism is doing. It forces you to articulate the reference — "what the code is supposed to do" — out loud, in words, before talking about what's actually on the screen. The duck doesn't solve your bug. The duck makes you build the answer key in real time, by speaking it. The bug usually fixes itself somewhere in the middle of the third sentence, because the third sentence is when you realize that your reference and your code don't actually match the assumption you've been clinging to.

I've started doing a written version of this with my own bot. Before I touch a failed scenario, I write a paragraph in plain English describing what the bot was trying to do at that moment, what each leg of the trade was supposed to produce, what the on-chain state was supposed to satisfy, and what the success criteria were. Only then do I open the logs. It's slower for the first ninety seconds and dramatically faster for the next ninety minutes.

The Two-Strike Heuristic

There's no canonical study that uses the phrase "two-strike rule" in debugging — I've gone looking. But the rule has hardened into a habit that the bot has, frankly, beaten into me. If I try to fix a problem twice and it isn't fixed, I stop touching code. Not pause. Stop. Close the editor, open the docs.

The reasoning is straightforward. The first failed fix can be bad luck. The second one means my mental model is wrong. If my mental model is wrong, every additional change I make is a coin flip — and the coin is weighted toward making things worse, because now there are more moving parts and less of a clean baseline. The 1986 CHI study found exactly this pattern in novices: they "frequently added bugs to the program during the course of trying to find the original one." The way out isn't to flip the coin harder. The way out is to stop, go read the reference, and rebuild the mental model.

Agans's Rule #1 says the same thing in different words. Tietz-Sokolskaya's Step 3 says it. MIT 6.102's "Don't Fix Prematurely" says it. They all converge on the same dull, repeatable, unsexy idea: when in doubt, the answer is not another patch. The answer is to leave the code alone and go re-establish what correct looks like.

This is the hardest discipline to keep because every cell in your body wants to keep typing. Typing feels like progress. Reading feels like surrender. It's the opposite. After enough cycles of patching patches, you start to recognize the feeling — that hot, narrow tunnel-vision that comes from being three guesses deep with no exit plan — and you can use it as a literal trigger. Tunnel vision means stop. Tunnel vision means open the docs. Tunnel vision means the reference is missing.

Building the Reference Before You Need It

The really useful version of this habit isn't reactive. It's proactive. The cheapest debugging is the debugging you do before the bug shows up, by building the reference into the project as you go.

For my bot, that means a few concrete habits that have nothing to do with code and everything to do with discipline.

For every external protocol the bot touches, I keep a short text file that summarizes, in my own words, what each instruction does, what its inputs mean, what its error codes signify, and which conditions can trigger each one. Not a copy-paste of the docs. A summary that I had to internalize enough to write. When the docs change or I learn something new, I update the file. When something breaks on chain, that file is the first thing I open. It's my pre-built answer key.

For every non-trivial path through the bot's own logic, I keep a paragraph in the project notes describing what the path is supposed to do at a state level — what the world should look like before the path runs, what should change, and what should be true after. When something fails inside that path, I read the paragraph first, then the logs. The paragraph is my own internal reference.

When I add a new integration, the first thing I write isn't the integration code. It's a manual walk-through, in prose, of what a correct interaction looks like end to end. Then I write a single happy-path test that asserts that interaction. The test, in effect, becomes the executable form of the reference. From that day forward, if the integration regresses, I have something to compare against without having to rebuild the mental model from zero.

This is what behavior-driven development was trying to teach the whole time, and I refused to listen for years because the BDD tooling looked baroque. The tooling is irrelevant. The discipline is what matters: define expected behavior in writing before implementation, and you've already done the most important debugging work of the project's life.

The Reference Beats the Stack Trace

There's a tempting fantasy in debugging that the stack trace will tell you the answer if you just stare at it long enough. It won't. The stack trace tells you where the program was when it noticed something was wrong. The stack trace is the substitute teacher pointing at the failed exam and saying "this is the page where the student got something wrong." Great. Which answer is wrong? The substitute teacher can't say.

Only the answer key can say. Only the reference.

This is true at the smallest scale and the largest. At the smallest scale, the reference is the docstring of the function you're calling — what does it promise to return, under what conditions? At the medium scale, the reference is the protocol's specification — what does this instruction actually do, what does it assume, what does it forbid? At the largest scale, the reference is the system's design — what is this whole thing for, what is it not for, and what are the invariants that must always hold?

When any one of those references is missing, debugging at that scale becomes a guessing game. When all of them are present, debugging becomes comparison. Comparison is fast. Guessing is slow. That's the entire trick. There is nothing else.

What This Changed About My Days

After I made reference-first my default move, three things changed about how I spend my time with the bot.

First, the bad nights got shorter. Not because debugging itself got faster on every bug — some bugs are still nasty — but because I stopped having the specific kind of bad night where I patched five things and ended up with six new bugs. The patches-on-patches death spiral happens almost exclusively when you don't have a reference. With one, you naturally stop after each change to ask "did this move the actual state toward the reference state, or away from it?" That question alone kills the spiral.

Second, I started writing more before writing more code. Not full design documents — short, scrappy paragraphs in plain English about what something is supposed to do. The paragraphs feel like they're slowing me down at the moment I write them. Two weeks later, when I'm staring at a failure in that subsystem, the paragraphs are the most valuable thing in the repo.

Third — and this is the one I didn't expect — I started enjoying debugging more. Debugging without a reference feels like being lost in a city with no map and no street signs. Debugging with a reference feels like a puzzle with a known solution. The bug is somewhere in the gap between the reference and the observation, and the work is to localize the gap. That's fun. That's the part of programming that hooked me in the first place.

The ACM Queue article I quoted earlier closed with a note about growth mindset — the Carol Dweck idea that people who believe their abilities can develop through effort outperform people who believe their abilities are fixed. Debugging, the article argues, is one of the most punishing tests of growth mindset in the entire field, because every bug is a small reminder that you got something wrong. Reference-first debugging is, in some ways, a coping mechanism for exactly that. It tells you, before you start, that the goal isn't to be right — the goal is to compare. The bug isn't a character flaw. It's a delta between the reference and the observation. Close the delta, ship the fix, move on.

The bot is still finding new ways to fail. That hasn't changed and probably never will. What's changed is that I now spend the first minute of every failure asking what the correct answer was supposed to be, before I spend a second on what the actual answer is. The first minute is what saves the next hour. Sometimes it saves the next week.

Key Takeaways

• Most failed debugging sessions are failed reference-gathering sessions in disguise — you can't identify a defect without first knowing what correct behavior looks like.
• Every major systematic debugging methodology (Agans, Zeller, MIT 6.102, Cornell CS312, Tietz-Sokolskaya) places "understand the system first" as rule one, not as a late-stage step.
• The 1986 ACM CHI expert-vs-novice study found experts solved bugs in roughly a third of the time and with a higher success rate, largely because they built a mental model of the system before touching code.
• A two-strike heuristic — after two failed fixes, stop touching code and go re-read the reference — prevents the patches-on-patches spiral that the novice study explicitly documented.
• Building the reference proactively (short prose summaries of intended behavior, happy-path tests as executable specifications) is cheaper than rebuilding it under pressure when something breaks at 2 a.m.