Weighted Scanning — Why I Stopped Inspecting Everything

The Day My Bot Got Slower the More I Added

There is a moment every infrastructure engineer hits at least once, and I am hitting it now. I keep adding pools to my bot's scan list, and the bot keeps getting slower. Not in a smooth, predictable way. In the way a Costco checkout line gets slower the closer you get to closing time — every additional cart does not just add its own time, it steals time from everybody else.

The instinct, when you start, is to scan more. More pools, more DEXes, more token pairs. If arbitrage is finding mispricings, then surely seeing more of the market means catching more of them. That is the intuition I started with, and it is the intuition that is wrong.

The number that finally broke it for me is from an academic paper I have been re-reading: an arXiv study of Uniswap V2 arbitrage between 2020 and 2023 found that filtering to the top 100 tokens by liquidity captures roughly 90% of total value locked, according to the arXiv paper 2406.16573. Ten percent of the surface, ninety percent of the opportunity. If I scan the other ninety percent of pools, I am paying full cost for the last ten percent of value — and on a chain where blocks come every four hundred milliseconds, I cannot afford to pay full cost for anything.

So this episode is about the thing I am learning to do, badly, in real time: weighted scanning. Not inspecting everything. Treating every pool, every path, every transaction as a candidate that has to earn the cycles I spend on it.

The Honest Math of an Exhaustive Scan

Let me describe the naive approach the way I actually built it first, because I think most beginners build the same thing and then have to unbuild it.

You subscribe to every pool you know about. You decode every account update. You enumerate every cycle of length two, three, maybe four through the resulting graph. You simulate each cycle. You rank by predicted profit. You submit the best one.

It is a clean architecture. It is also a non-starter against any competitive bot, and the reasons compound on top of each other.

The first reason is simulation cost. An open-source bot post is candid about this: an online simulation — calling out to a node to dry-run a transaction — runs at "approximately 0.3–1.0 second latency per call," according to the blog post. On a chain with sub-second blocks, one online simulation already exceeds a block. Ten of them, and you are watching the market from yesterday.

The second reason is detection latency. An infrastructure write-up publishes a table that should be hung on the wall of every bot developer: under thirty milliseconds of round-trip latency, capture rate sits in the eighty to ninety percent range. Above two hundred milliseconds, capture rate drops below ten percent, per an MEV infrastructure blog. The latency budget is the size of a slow blink. Every cycle you spend simulating a path you would never have submitted is a cycle that pushes the path you would have submitted past the cliff.

The third reason is the most uncomfortable: the market is getting more efficient over time. The same arXiv paper documents an arbitrage profit decline of roughly 0.002 daily slope across its three-year window, which the authors describe as "market mechanisms…being arbitraged more efficiently," per the arXiv paper. The naive scan worked better in 2021 than it works now. Selectivity is not a luxury — it is the direction the river is flowing.

Weight One: Liquidity Earns the First Look

The simplest weight, and the one I am implementing first, is liquidity. A pool with a few thousand dollars of TVL has a different role in my life than a pool with tens of millions. Both can produce arbitrage opportunities, but the small pool's opportunities are usually too small to clear gas plus fees, and they evaporate the moment a real trade hits them.

The arXiv paper validates this rough rule: filtering its experimental dataset to the top 100 tokens by TVL — pools with more than $20,000 of liquidity — captures roughly 90% of total Uniswap V2 value, per the arXiv paper. It is the Pareto principle showing up in the most literal possible form. Ten percent of the surface, ninety percent of the opportunity.

This is the part that always feels like cheating. You skip the long tail and you assume nothing valuable lives there. But "long tail" in DEX terms means "the place where you cannot exit your position without moving the market against yourself." Even when the spread looks juicy, the slippage on the exit usually eats the spread. That is not pessimism. That is the AMM curve doing what AMM curves do.

So my first weight is a hard liquidity floor. Below the floor, the pool exists in my data structures but never gets simulated. It is on the bench. If liquidity grows past the threshold, it gets called up. This single change reduced the number of paths my prototype evaluates per cycle by something I would call dramatic — the kind of reduction where the bot's runtime stops looking like a flat line trending upward and starts looking like the line you actually want.

Weight Two: Recent Activity Is a Live Wire

The second weight, and the one I find more interesting, is recency. A pool that has just been touched by a swap is a different animal than a pool that has been quiet for an hour.

This is the insight behind the entire event-driven design pattern. Polling means you ask every pool, on a schedule, whether anything has happened. Subscription means the pool tells you the moment something happens. The data for this is striking: the pipeline goes from roughly five hundred milliseconds on a polling design to under fifty milliseconds on an event-driven design, per an MEV infrastructure blog. That is an order of magnitude, and it comes from doing less, not more.

But the deeper point is that recency is itself a weight signal. When a pool emits a swap event, two things just became true. First, somebody believed there was reason to trade in that pool right now. Second, the pool's price just moved relative to wherever it was before. Both of these are predictors that an arbitrage cycle through that pool might be live, in a way that a quiet pool's cycle is almost certainly not.

So the second weight in my system is a freshness score. A pool that was touched in the last block gets a high score. A pool quiet for ten blocks decays. A pool quiet for a thousand blocks is functionally archived — it stays in the graph because the topology might matter again later, but I do not pay the cost of running cycle detection through it on every tick.

The NASCAR pit-crew analogy is the one that finally made it click for me. A crew chief does not check every gauge on every lap. They watch the gauges that just changed. The cars they care about most are the ones whose situation just shifted — fresh tires, fresh fuel, fresh damage. Quiet cars in the middle of the pack are not worth a glance until something happens to them.

Weight Three: Structural Position in the Graph

The third weight is the one I am still working out, and it is the most subtle. It is structural. Some pools are connectors and some are dead ends.

The arXiv paper introduces what it calls a line graph transformation: pools become vertices, and the connections between them encode shared tokens. In this representation, the question "is there an arbitrage cycle?" becomes "is there a negative-weight cycle in the line graph?" — and the algorithm to find it is a modified Bellman-Ford variant the authors call MMBF, per the arXiv paper.

The weight on each edge in this graph is −log((1−λ) × r_j / r_i), where the negative logarithm turns multiplicative price ratios into additive path costs. A profitable cycle becomes a path with negative total weight. The math is elegant enough that I find myself sketching it on whatever scrap of paper is nearby.

What strikes me about this representation is what it implies for prioritization. A pool that participates in many cycles — a stablecoin pool, a SOL pair, a high-liquidity routing hub — is structurally more valuable than a pool that participates in one. When that hub pool's price moves, dozens of potential cycles change at once. When a leaf pool's price moves, maybe one cycle changes, and that cycle goes through a connector you would have re-evaluated anyway.

The paper's results are dramatic enough that I want to repeat them here. Their MMBF approach identified 23,868 arbitrage paths exceeding $1,000 of profit, compared to 19 such paths from a baseline algorithm, per the arXiv paper. That is not a marginal improvement. That is a different category of system. And the gain comes not from scanning more — it comes from a smarter representation that surfaces the paths that matter and ignores the ones that do not.

For my bot, this translates to something I can actually implement: rank pools by structural connectivity, and when a connector pool moves, prioritize cycles that pass through it. Hub pools earn more cycles. Leaf pools earn fewer. Same Bellman-Ford machinery, but the weights at the front determine which subgraph I run it on.

Weight Four: Profit Threshold as the Final Gate

The last weight is the one that runs at the end of the pipeline, and it is the simplest: a path that does not predict enough profit to justify the gas plus the tip plus the risk does not get submitted.

A guide is direct about the numbers production bots use as gates: minimum slippage tolerance of ten basis points, minimum predicted profit of ten basis points, and a final on-chain profit check at five basis points before the transaction commits, per the guide. Below those thresholds, the path is dropped. Not retried, not re-simulated with different parameters — dropped.

This seems obvious, but it is structurally important. Without this final gate, every previous optimization gets undermined by a flood of low-margin submissions that fail more often than they succeed and pay tip fees regardless. The same guide notes that bots typically pay 50–60% of expected profit in priority tips on Solana, per the guide. If your edge is thin, the tip eats it. The threshold gate is what stops you from paying tips on paths whose edge was never real to begin with.

Profitability test results illustrate the same boundary from the bottom up. With 100 USDT of input, the test bot produced 0.149 USDT of profit. At 400 USDT, profit peaked at 0.355 USDT. At 900 USDT, the position lost 0.101 USDT — the slippage from oversizing exceeded the spread, per the blog post. The profitability curve is concave, and the threshold gate has to find its peak, not just any positive number.

The Cost of Doing It the Other Way

The alternative to weighted scanning is not theoretical. It exists, and you can measure what it costs.

The Solana network's transaction reversion data tells the story plainly. A report on the Solana MEV ecosystem cites a peak in April 2024 where roughly 75.7% of non-voting transactions reverted, per the report. Three out of four bot transactions, on the worst day, were doing nothing but burning compute and fees. That is what an exhaustive, undisciplined scan-and-submit pipeline produces at scale: a network mostly busy failing.

The number went down after a scheduler upgrade in May 2024, per the report, but the underlying lesson did not. Reversion rate is a direct measurement of how many of your submitted opportunities were never opportunities. It is the cost of pretending a path is worth simulating when it is not, summed across the network.

On other chains the pattern is similar. A cross-chain analysis describes Optimism searchers sending "the same transaction 50 times" hoping that one will land, per the article. That is the brute-force approach in its purest form, and the same source notes Optimism's success rates land in the 10–50% range while Ethereum's bundle approach reaches roughly 99%. The chain that selects what to submit beats the chain that submits everything, by an order of magnitude.

What This Looks Like as a Pipeline

If I draw the system the way I am building it now, the weights compose into a funnel.

At the top of the funnel sits the subscription layer. I am not subscribing to every account on the chain — I am subscribing to a curated set of pool accounts that pass my liquidity floor. The subscription protocol matters; a guide on Solana subscription makes the general point that polling adds latency compared to streaming, per the guide. Each protocol has different characteristics — choose based on use case.

Below the subscription layer is the freshness filter. An update arrives, and I check whether the pool that just changed is one whose movement would plausibly affect a high-value cycle. If not, the update is recorded but not acted on.

Below that is the cycle generation step. For pools that pass the freshness filter, I enumerate cycles weighted by structural position — connector pools get priority, leaf pools get evaluated only when the math makes them obviously worth it.

Below that is local simulation. The guide notes that local simulation, done well, can stay under a millisecond per path, per the guide. At that speed, I can afford to simulate the candidates that survive the filters above, but only those.

At the bottom of the funnel is the profit threshold gate, the bisection-style optimization for input size, and submission. By the time a path reaches submission, it has survived four independent weight checks. The expected reversion rate, if all four weights are calibrated, drops far below what an exhaustive pipeline would produce.

None of this is finished. The weights are wrong, the thresholds are guesses, and I expect to be re-tuning them for months. But the shape of the system is right, and I can feel it in the runtime numbers in a way I could not feel the naive scan.

The Mental Model That Replaced "Check Everything"

What changed for me, doing this, is the mental model.

I used to think of the bot's job as searching. The bigger the search, the more thorough the bot. The slow path was the safe path because it would not miss anything. The trouble was that thoroughness, in this domain, is indistinguishable from being late, and being late is indistinguishable from being wrong.

The new model is closer to how an air traffic controller works. You are not watching every plane in the sky. You are watching the planes whose situation just changed, and you are routing your attention by a priority that updates faster than the planes do. The radar shows everything; your attention does not. That is not laziness. That is the only way the job gets done.

In arbitrage terms: the chain shows everything; my bot's attention does not. The pools I do not simulate are not pools I have given up on. They are pools that have not yet earned the cycles. When they earn them — by gaining liquidity, by being touched, by sitting on a path through a hub that just moved — they get evaluated. Until then, they wait their turn.

And the corollary, which I keep having to remind myself of, is that the right size of "everything" is not as big as I think it is. The arXiv result keeps pulling me back to the same conclusion: the top of the distribution is where the value lives, and the long tail is mostly noise dressed up as opportunity, per the arXiv paper. The discipline of weighted scanning is the discipline of trusting that math instead of fighting it.

Where I Am, Right Now

I do not know yet how well this will work. The thresholds I have picked are first guesses, and I expect the calibration phase to be longer than the implementation phase. I do not know which weight matters most. I do not know how the weights interact when the market is quiet versus when it is volatile. I do not know whether I have built a smarter system or just a more complicated one.

What I do know is that the naive scan was a wall. Every additional pool made it slower, and "slower" on a four-hundred-millisecond block is a synonym for "already lost." The weighted approach at least gives me a system whose costs scale with what matters, not with what is theoretically possible. That is not the same as profitability. But it is the only architecture in which profitability is even reachable.

The rest is calibration, which is the rest of the project.

Key Takeaways

• Naive exhaustive scanning fails because simulation cost and detection latency budgets are tight; under thirty milliseconds of round-trip latency yields 80–90% capture, while above two hundred milliseconds it falls below ten percent (source).
• Liquidity weighting is the highest-leverage filter: roughly the top 100 tokens by TVL captured about 90% of Uniswap V2 value in the arXiv dataset (arXiv 2406.16573).
• Recency weighting plus event-driven subscriptions cut a typical pipeline from roughly five hundred milliseconds to under fifty milliseconds (source).
• Smarter graph representations beat brute force; the MMBF algorithm surfaced 23,868 paths above $1,000 profit versus 19 for a baseline approach in the same dataset (arXiv 2406.16573).
• A profit threshold gate is non-negotiable; without it, tip costs that typically run 50–60% of expected profit on Solana eat any thin edge a path was carrying (source).