How to Spot a Fake Edge (Before It Costs You)
Three traps that make a losing idea look like a winner and the live test I'm running to avoid all three.
Every day someone shows you a model that “works.” A backtest, a screenshot, a track record. Most of them are fooling you and the honest ones are fooling themselves.
Here are the three traps that make a fake edge look real, how to catch them in anyone’s results, and how I’m holding my own live experiment to all three including the part where it’s currently losing.
Trap 1 — Counting days instead of bets.
“It cleared significance!” sounds like a verdict. It usually isn’t. What matters isn’t how many days of data someone has, it’s how many independent outcomes. A genuinely real edge, measured over a short stretch, will often look weak. A fake one will often look strong by pure luck.
But there’s a sneakier version, and it’s the one that got me: a real signal can look far stronger than it is because the current market happens to be shaped like your signal. That’s my situation right now. My contrarian signal isn’t so much beating the market as the market is temporarily bending toward it. When that regime passes, so might the edge.
The formal version of "how much independent data before a result means anything" is old news. Andrew Ng walks through it in CS229, Lecture 9 (Bias/Variance & sample complexity). If someone can't tell you how many independent bets sit behind their number or whether it only held in one kind of market the number doesn't mean what they think it does.
Trap 2 — Moving the finish line.
A subtle one that trips up even honest people. Watch any noisy result every day, announce the moment it looks good, and you’ve cheated without meaning to. A number that wanders will eventually cross the “success” line if you keep checking. The only fair test names its finish line in advance and reports whatever it finds there good, bad, or nothing.
Trap 3 — The question they hope you won't ask.
Someone shows you a signal with a gorgeous correlation. Here's the question that detonates most of them: how many did you try before you found that one? Test twenty ideas and one will look brilliant by chance alone. The only honest number is the one measured on data collected after you committed to the signal, never the data you used to pick it.
So here's my test, held to all three — in public:
One signal, one finish line, both fixed before the clock started.
I don’t get to call it the first day it looks good. It has to clear the bar and hold it across the independent windows.
The only number I’ll ultimately stand on is the forward one, measured after I committed, immune to the “how many did you try” trap.
And the uncomfortable part: the main model is losing right now. It’s been reliably, measurably wrong for weeks. I’m showing you anyway because a test you only publish when it flatters you isn’t a test. It’s marketing.
That’s the whole experiment: a real edge, proven under rules designed to make it hard to fool you or an honest failure, in the open. I already know which outcome most people would hide. I’m going to show you either one.
Follow along and watch which it turns out to be.
The free posts are the scoreboard and the rules. If you want the game film the specific names my two competing signals fight over each day, what's breaking under the hood, and the real-time autopsy as this plays out, that's what I'm opening up for paid readers. Not tips. A front row seat to a real experiment, honestly run. (I'm a builder sharing research, not a financial advisor.)