Config File Defaults: The Trap That Bites You in Production

The Bug That Only Exists in Production

I am sitting in front of the screen at midnight, watching the bot do something it absolutely should not be doing. On my development laptop, the same code ran clean for hours. On the production box, it is making decisions I never told it to make.

The gap between the two environments is not the network, not the RPC endpoint, not the wallet. The gap is a single file: a YAML config that lives next to the binary in production and does not exist on my laptop. Two values inside that file disagree with the defaults I wrote in code. And because production reads the file and development reads the code, the two machines are running, in effect, two different bots.

This is the kind of bug that does not show up in unit tests. It does not show up in staging if your staging uses the dev config. It only shows up the moment money is on the line. I want to spend this post walking through what I learned in the hours after I noticed, and what the broader software engineering community has written about the same trap — because it turns out almost nobody escapes this one.

Why a MEV Bot Cares So Much About Config

Before the bug, let me explain why a Solana arbitrage bot has a config file at all. The bot has dozens of knobs that are not algorithm-defining but are profit-defining. Things like minimum profit thresholds, scan intervals, retry counts, the priority fee strategy, which DEX programs to include in path search, how aggressively to expand the cycle graph. None of these are constants in the mathematical sense. They are tuning parameters.

If I hardcode them, every change is a recompile and a redeploy. If I put them in environment variables, the list grows past anything I can remember without scrolling. So like most production systems, the bot reads a config file at startup, falls back to defaults when keys are absent, and treats the merged result as the operating envelope for the run.

That last sentence — "falls back to defaults when keys are absent" — is where the snake lives.

The Anatomy of the Trap

Here is the pattern, stated plainly, because once you see it you will see it everywhere:

1. The code declares a default for some parameter — say, a minimum profit threshold.
2. The production config file declares the same parameter, but with a different value.
3. In development, the config file is empty, partial, or missing, so the code default applies.
4. In production, the config file is present and complete, so the file's value applies.
5. The two environments now disagree, and the developer has no idea, because both environments look like they are "using defaults."

The word "default" is doing a lot of work here, and it is doing it dishonestly. There are actually two defaults — one in code, one in the file — and the runtime picks whichever it finds first. Which one wins depends on something as mundane as whether the YAML key is present.

Robat Williams at Scott Logic published a short, sharp essay on this exact problem back in 2018. His definition, lifted from the Oxford English Dictionary, is the cleanest framing I have read: a default is "a preselected option adopted by a computer program when no alternative is specified." The problem, he argues, is that when a default value is used, there is usually no record anywhere in the code of that decision being made. Future maintainers cannot tell whether the absence of a value is intentional or an oversight (Scott Logic).

In my case I had two defaults disagreeing, but the underlying disease is the same: invisibility. Nothing in the code or the config file said "this value is intentional and should match the other one." The two sides had drifted, silently, over weeks of small commits.

How I Tracked It Down

The symptom was that the bot was firing on opportunities I expected it to skip. Either the profit gate was looser in production, or some upstream filter was set differently. I started by dumping the effective config at startup — printing every key the bot actually used, including the ones that were filled in from defaults.

The dump revealed the discrepancy in about ninety seconds. The YAML file specified a slightly looser threshold than the in-code default. Looking at git blame, I had tuned the YAML during a tuning session weeks earlier and forgotten to mirror the change back into the code. Development, lacking the YAML override, kept running with the tighter code default. Production used the looser YAML value. Same binary, two behaviors.

The fix, in the mechanical sense, took thirty seconds. The fix in the deeper sense — "how do I make sure this never happens again" — took the rest of the night, and led me down a rabbit hole of how the rest of the industry handles config.

The Industry Has Mostly Bled For This Lesson

It turns out my bug has a long and embarrassing lineage. The closest match I found was a write-up by Nick Janetakis, a solo SRE who refactored a CodeIgniter project's configuration by moving shared variables into a common base file. The production environment-specific file ended up with nothing but comments and conditional statements — nearly empty. Because CodeIgniter 3 expected at least one variable assignment in every environment-specific file it loaded, the production app failed to merge config correctly and refused to start. Development and test, with complete config files, passed cleanly. Only production failed, and only Kubernetes health checks kept the failure from reaching real traffic (Nick Janetakis).

His takeaway is one I am writing on a sticky note: "Do not blindly trust that your staging environment will catch everything." Staging is only as good as the inputs you feed it. If staging runs against a different config file than production, staging is not staging — it is a louder dev environment.

A DEV Community post by an author writing as "sebm" makes the matching point about silent fallbacks. The author argues that silently falling back to default configuration creates false confidence and masks misconfiguration errors. A concrete example they give: a typo like USER_IMGAES_BUCKET_NAME goes undetected because the app silently substitutes a default bucket. The app appears to run. It writes user images to the wrong place. Hours of debugging follow (DEV Community).

The recommendation across all of these sources converges on a single principle: fail fast when a required value is missing. Do not paper over the absence. Throw, log loudly, refuse to start. The cost of a noisy failure at boot is always cheaper than the cost of a silent misbehavior in production.

The Twelve-Factor App Has Opinions About This

The canonical text on this topic is the config section of the Twelve-Factor App methodology, written by Adam Wiggins. The headline rule is "store config in the environment," by which Wiggins means environment variables rather than files checked into source control. His litmus test is striking in its simplicity: "whether the codebase could be made open source at any moment, without compromising any credentials" (12factor.net).

If the answer is no — if you would lose secrets the second you flipped the repo public — then your config is in the wrong place. By that test, my YAML-with-defaults approach is leaky around the edges. The file does not contain credentials, but it does contain operational tuning that I would not want broadcast. I have been treating it as code-adjacent, when by twelve-factor logic it should be environment-adjacent.

There is a sharper twelve-factor opinion that I have been ignoring: env vars should not be grouped into named environments like "development" or "production." Wiggins argues that this grouping causes a combinatorial explosion as the number of deploy environments grows. A single env var like LOG_LEVEL=info should be set independently per deploy, not pulled from a development.yaml block.

I am not sure I fully agree for a single-developer MEV bot — the YAML structure gives me a readable place to keep tuning history. But the warning is real. The more environments I add, the more places defaults can drift apart.

YAML Has Its Own Special Traps

There is a thoughtful post on aran.dev titled "yaml configuration: the best of a bad bunch" that argues for a hybrid approach: keep YAML for structure and readability, but let values reference environment variables with a syntax like ${ENV_NAME:default} (aran.dev). The advantage is that the default lives in exactly one place — the YAML file — and the override path is explicit.

The author also describes a real production bug they encountered: a dollar sign appearing inside a config value triggered their custom environment-variable parser unexpectedly. They had to retract the simpler $ENV_VAR syntax and recommend only the safer ${ENV_VAR:default} form. This is the kind of detail that gets you the next time. Anywhere config goes through a parser, the parser can become a hidden source of surprise.

Even mainstream frameworks have not escaped this. The Spring Boot project has a documented inconsistency, tracked as GitHub Issue #38969, where certain values cannot be overridden via YAML but can be overridden via the classic application.properties file (GitHub). A developer who tries to override a default in YAML, sees it not take effect, and assumes their syntax is wrong is having a frustrating evening — when in fact the framework simply does not honor that override path uniformly.

The meta-lesson is that the mechanism by which a default gets overridden is itself part of the contract, and that contract is rarely written down clearly.

Concrete Numbers From Real Deployments

For anyone who wants to see what a sensible dev-versus-prod config diff actually looks like, OneUptime published a January 2026 piece that lays out a Kubernetes Kustomize-style overlay (OneUptime). Their examples are reasonable defaults to think about:

• LOG_LEVEL: info in development, warn in production. Dev wants noise for debugging; prod wants quiet so the logs do not become a haystack hiding the needle.
• CACHE_TTL: 300 seconds in development versus 3,600 seconds in production. Dev wants fast invalidation so code changes are visible; prod wants longer caching for performance.
• DATABASE_HOST: a local placeholder in dev, the actual cluster hostname in prod.
• DB connection pool size: a documented default of 10 with a valid range of 1 to 100.

A detail that caught my attention from the same piece: "By default, environment variables from ConfigMaps are not updated when the ConfigMap changes. Pods must be restarted." This is one of those Kubernetes facts that experienced operators internalize and newcomers learn the hard way. You edit the ConfigMap, you confirm the new value in the cluster, you watch the running pod do nothing different. The new value is sitting there. The pod is reading the old value. Nothing changes until you restart.

This is the exact same trap as my YAML bug, just at a different layer. A change has been made in one place. The runtime has not noticed. The longer the lag goes undetected, the more confused everyone gets.

What "Good Defaults" Actually Look Like

Pulling from the same Scott Logic essay, here are the properties Williams says a good default should have. I am going to repeat them not because the list is exotic, but because every one of them maps directly onto a bug I have personally caused.

1. Safe. Defaults must not cause insecure behavior or irreversible actions. Because defaults get used accidentally, the cost of an unsafe default is paid at the worst possible moment.
2. Expected. The default should not surprise the reader. If you have to explain it, it is probably wrong.
3. Consistently applied. A default should not change based on context. Two callers asking for "the default" should get the same answer.
4. Backwards compatible. When you upgrade software, existing behavior should not change unless the user opts into the new behavior.
5. Visible. Without a record that a default was used, future maintainers cannot tell whether the choice was deliberate. This is the heart of my bug.
6. Explicitly specified with comments. Williams's preferred fix is to write the default out in the call site even when it would be inferred, and add a comment explaining why that value.

If I had followed point six — write out the default in the YAML even when it matches the code default, with a comment — my drift would have been visible the moment the YAML and the code disagreed. The bug would have shown up in code review, not in production at midnight.

What I Changed In My Bot

The immediate fix was to mirror the values: the YAML's defaults now match the code's defaults, with comments on both sides pointing at each other. But that is a band-aid; the same drift will happen again the moment I tune one without the other.

The deeper fix is a startup ritual the bot now performs every time it boots:

• Dump the effective config. Every key, every value, every source — whether it came from the YAML file, an environment variable, or a code default. This goes to the log in the first second of the run. If anything is off, I see it before the bot does anything financial.
• Fail fast on missing critical values. The bot no longer silently substitutes a default for anything that affects money. If a key is missing that has any influence on trade execution, it refuses to start and prints exactly which key was missing.
• Treat dev and prod symmetrically. Where I used to let dev run without a config file, I now require both environments to load a file. Two files with the same shape, different values. Either environment that boots without a file is treated as a misconfiguration.
• Document the valid range for every numeric. Borrowing the OneUptime convention, every tunable has a comment noting its sensible range. The minimum profit threshold is not just a number — it is a number with a unit, a valid range, and a one-sentence rationale.

None of this is exotic. It is the engineering discipline that every mature SaaS team converges on, often after exactly the kind of incident I described. The reason I am writing it down here is that nobody told me, and I had to learn it from a private outage. Maybe someone reading this gets to skip a few hours of staring at a YAML file at 2 a.m.

The Broader Pattern

If I zoom out, this whole episode is a specific case of a general failure mode: two sources of truth for the same value, with no mechanism to keep them in sync. The same pattern shows up in cached data versus its source, in client-side validation versus server-side validation, in documentation versus implementation. Anywhere a value lives in two places, the two places will eventually disagree, and you will only find out when production behaves wrong.

The engineering response, every time, is to either eliminate one of the sources or to make the disagreement visible. Eliminate by having one place define the value and the others reference it. Make visible by logging the effective merged state at the moment it is computed, so a human can spot the drift.

For config files, I think the eliminating approach is the right one in the long run. The code should not have its own default if the config file has one. The config file should be required, not optional. The merge should be deterministic and logged. There should be no "fallback to baked-in default" path in code that handles real money.

Is that always practical? No. Sometimes you want a working dev environment without a config file at all. But the more the code path can be made identical between dev and prod, the fewer of these subtle disagreements you have to chase.

What This Means Going Forward

The deeper realization is that config management is part of the system's correctness, not a sidecar to it. I had been treating the YAML as a convenience — a place to stash numbers so I would not have to recompile. But the moment the values in that file affect what the bot trades, the file is part of the trading logic. It deserves the same review discipline as code: pull requests, diffs, comments explaining why values changed, an explicit owner.

If I keep tuning the bot the way I have been — adjusting values in a YAML, redeploying, watching the result — eventually one of those tweaks will be a regression I do not catch for days. Adding the startup dump and the fail-fast checks buys me time. Treating the file as code closes the gap further.

For anyone building a production system with a tuning surface this wide, my suggestion is to do this work before you have your own midnight YAML story. The cost of setting up effective-config logging on day one is almost zero. The cost of debugging a silent default drift after the bot has been running for weeks is a long, very specific kind of pain.

Key Takeaways

• Two defaults are worse than one. When code defaults and config file defaults both exist, they will eventually disagree, and the runtime will pick one without telling you.
• Silent fallbacks lie. Substituting a default for a missing or typo'd key gives the false impression that the system is configured correctly. Fail fast instead.
• Log the effective config at startup. Every key, every value, every source. Make merged state inspectable in the first second of the run.
• Treat the config file as code. It belongs in version control, in code review, in the same discipline as everything else that affects production behavior.
• Do not trust staging to catch config drift. Staging is only as good as its inputs; if it does not load the production config, it is not testing production.